Nov 03 20:36:30.910881 ip-10-0-128-175 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 03 20:36:30.910890 ip-10-0-128-175 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 03 20:36:30.910897 ip-10-0-128-175 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 03 20:36:30.911129 ip-10-0-128-175 systemd[1]: Failed to start Kubernetes Kubelet. Nov 03 20:36:41.001648 ip-10-0-128-175 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 03 20:36:41.001700 ip-10-0-128-175 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 6840dd11ad1f47088cfc1d96c63aaed5 -- Nov 03 20:38:44.583734 ip-10-0-128-175 systemd[1]: Starting Kubernetes Kubelet... Nov 03 20:38:44.965848 ip-10-0-128-175 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 20:38:44.965848 ip-10-0-128-175 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 20:38:44.965848 ip-10-0-128-175 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 20:38:44.965848 ip-10-0-128-175 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 20:38:44.969597 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969513 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:38:44.969597 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969597 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:38:44.969597 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969601 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:38:44.969694 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969605 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:38:44.969694 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969608 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:38:44.969694 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969611 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:38:44.969694 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969614 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:38:44.969694 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969616 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:38:44.969694 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969619 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:38:44.969694 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969621 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:38:44.969694 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969624 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:38:44.969694 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969627 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:38:44.969694 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969629 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:38:44.969694 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969632 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:38:44.969694 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969634 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:38:44.969694 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969637 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:38:44.969694 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969640 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:38:44.969694 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969642 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:38:44.969694 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969648 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:38:44.969694 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969651 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:38:44.969694 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969654 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:38:44.969694 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969656 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:38:44.969694 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969659 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:38:44.970199 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969662 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:38:44.970199 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969664 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:38:44.970199 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969667 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:38:44.970199 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969669 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:38:44.970199 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969672 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:38:44.970199 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969675 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:38:44.970199 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969680 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:38:44.970199 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969684 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:38:44.970199 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969688 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:38:44.970199 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969692 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:38:44.970199 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969694 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:38:44.970199 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969697 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:38:44.970199 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969701 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:38:44.970199 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969704 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:38:44.970199 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969708 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:38:44.970199 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969712 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:38:44.970199 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969715 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:38:44.970199 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969718 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:38:44.970199 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969720 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:38:44.970703 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969723 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:38:44.970703 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969725 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:38:44.970703 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969728 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:38:44.970703 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969730 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:38:44.970703 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969733 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:38:44.970703 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969735 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:38:44.970703 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969738 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:38:44.970703 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969740 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:38:44.970703 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969744 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:38:44.970703 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969747 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:38:44.970703 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969749 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:38:44.970703 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969752 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:38:44.970703 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969754 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:38:44.970703 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969757 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:38:44.970703 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969760 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:38:44.970703 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969763 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:38:44.970703 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969766 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:38:44.970703 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969768 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:38:44.970703 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969771 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:38:44.971225 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969773 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:38:44.971225 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969776 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:38:44.971225 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969779 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:38:44.971225 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969782 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:38:44.971225 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969784 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:38:44.971225 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969786 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:38:44.971225 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969789 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:38:44.971225 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969792 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:38:44.971225 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969794 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:38:44.971225 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969797 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:38:44.971225 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969800 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:38:44.971225 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969802 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:38:44.971225 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969805 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:38:44.971225 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969807 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:38:44.971225 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969809 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:38:44.971225 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969812 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:38:44.971225 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969815 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:38:44.971225 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969817 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:38:44.971225 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969820 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:38:44.971722 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969823 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:38:44.971722 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969825 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:38:44.971722 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969829 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:38:44.971722 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969832 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:38:44.971722 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969834 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:38:44.971722 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969837 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:38:44.971722 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969839 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:38:44.971722 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969842 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:38:44.971722 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969845 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:38:44.971722 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969848 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:38:44.971722 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969850 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:38:44.971722 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969853 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:38:44.971722 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969855 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:38:44.971722 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969859 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:38:44.971722 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969862 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:38:44.971722 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969864 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:38:44.971722 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969866 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:38:44.971722 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969869 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:38:44.971722 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969871 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:38:44.971722 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969874 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:38:44.972317 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969876 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:38:44.972317 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969879 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:38:44.972317 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969881 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:38:44.972317 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969884 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:38:44.972317 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969886 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:38:44.972317 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.969888 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:38:44.972317 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970689 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:38:44.972317 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970696 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:38:44.972317 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970699 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:38:44.972317 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970703 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:38:44.972317 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970706 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:38:44.972317 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970708 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:38:44.972317 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970714 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:38:44.972317 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970717 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:38:44.972317 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970720 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:38:44.972317 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970722 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:38:44.972317 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970726 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:38:44.972317 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970728 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:38:44.972317 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970731 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:38:44.972317 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970734 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:38:44.972859 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970737 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:38:44.972859 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970741 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:38:44.972859 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970745 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:38:44.972859 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970749 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:38:44.972859 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970752 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:38:44.972859 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970755 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:38:44.972859 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970757 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:38:44.972859 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970761 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:38:44.972859 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970763 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:38:44.972859 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970766 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:38:44.972859 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970769 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:38:44.972859 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970771 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:38:44.972859 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970774 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:38:44.972859 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970777 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:38:44.972859 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970779 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:38:44.972859 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970782 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:38:44.972859 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970784 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:38:44.972859 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970787 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:38:44.972859 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970789 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:38:44.973342 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970791 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:38:44.973342 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970794 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:38:44.973342 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970796 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:38:44.973342 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970799 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:38:44.973342 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970801 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:38:44.973342 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970807 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:38:44.973342 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970810 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:38:44.973342 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970813 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:38:44.973342 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970816 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:38:44.973342 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970819 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:38:44.973342 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970822 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:38:44.973342 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970825 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:38:44.973342 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970827 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:38:44.973342 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970830 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:38:44.973342 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970832 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:38:44.973342 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970835 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:38:44.973342 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970837 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:38:44.973342 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970840 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:38:44.973342 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970843 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:38:44.973879 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970845 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:38:44.973879 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970847 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:38:44.973879 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970850 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:38:44.973879 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970852 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:38:44.973879 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970855 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:38:44.973879 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970857 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:38:44.973879 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970859 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:38:44.973879 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970862 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:38:44.973879 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970864 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:38:44.973879 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970867 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:38:44.973879 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970869 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:38:44.973879 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970872 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:38:44.973879 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970874 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:38:44.973879 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970877 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:38:44.973879 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970879 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:38:44.973879 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970881 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:38:44.973879 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970884 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:38:44.973879 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970887 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:38:44.973879 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970890 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:38:44.973879 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970893 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:38:44.974396 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970896 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:38:44.974396 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970898 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:38:44.974396 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970900 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:38:44.974396 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970903 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:38:44.974396 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970906 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:38:44.974396 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970909 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:38:44.974396 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970911 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:38:44.974396 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970913 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:38:44.974396 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970916 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:38:44.974396 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970937 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:38:44.974396 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970941 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:38:44.974396 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970944 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:38:44.974396 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970947 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:38:44.974396 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970950 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:38:44.974396 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970952 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:38:44.974396 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970955 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:38:44.974396 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970957 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:38:44.974396 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970959 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:38:44.974396 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970962 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:38:44.974396 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970965 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:38:44.974923 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970968 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:38:44.974923 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970970 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:38:44.974923 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970972 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:38:44.974923 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970975 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:38:44.974923 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970977 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:38:44.974923 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970980 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:38:44.974923 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970982 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:38:44.974923 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970984 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:38:44.974923 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970988 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:38:44.974923 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970992 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:38:44.974923 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970996 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:38:44.974923 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.970998 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:38:44.974923 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971000 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:38:44.974923 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971003 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:38:44.974923 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971510 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:38:44.974923 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971517 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:38:44.974923 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971520 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:38:44.974923 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971523 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:38:44.974923 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971527 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:38:44.975409 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971531 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:38:44.975409 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971544 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:38:44.975409 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971547 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:38:44.975409 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971550 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:38:44.975409 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971553 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:38:44.975409 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971555 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:38:44.975409 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971558 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:38:44.975409 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971560 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:38:44.975409 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971563 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:38:44.975409 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971565 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:38:44.975409 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971568 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:38:44.975409 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971571 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:38:44.975409 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971573 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:38:44.975409 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971576 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:38:44.975409 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971578 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:38:44.975409 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971581 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:38:44.975409 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971583 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:38:44.975409 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971586 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:38:44.975409 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971588 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:38:44.975409 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971591 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:38:44.975928 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971593 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:38:44.975928 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971596 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:38:44.975928 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971600 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:38:44.975928 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971604 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:38:44.975928 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971606 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:38:44.975928 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971609 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:38:44.975928 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971611 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:38:44.975928 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971614 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:38:44.975928 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971616 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:38:44.975928 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971619 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:38:44.975928 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971622 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:38:44.975928 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971625 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:38:44.975928 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971627 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:38:44.975928 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971630 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:38:44.975928 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971632 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:38:44.975928 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971635 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:38:44.975928 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971637 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:38:44.975928 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971640 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:38:44.975928 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971642 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:38:44.976411 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971645 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:38:44.976411 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971647 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:38:44.976411 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971650 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:38:44.976411 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971652 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:38:44.976411 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971655 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:38:44.976411 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971657 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:38:44.976411 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971661 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:38:44.976411 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971663 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:38:44.976411 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971666 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:38:44.976411 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971668 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:38:44.976411 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971671 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:38:44.976411 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971673 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:38:44.976411 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971676 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:38:44.976411 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971678 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:38:44.976411 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971681 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:38:44.976411 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971684 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:38:44.976411 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971689 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:38:44.976411 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971691 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:38:44.976411 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971693 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:38:44.976411 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971696 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:38:44.976928 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971698 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:38:44.976928 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971701 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:38:44.976928 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971703 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:38:44.976928 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971706 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:38:44.976928 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971709 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:38:44.976928 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971711 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:38:44.976928 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971714 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:38:44.976928 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971716 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:38:44.976928 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971719 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:38:44.976928 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971721 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:38:44.976928 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971724 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:38:44.976928 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971727 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:38:44.976928 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971729 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:38:44.976928 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971732 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:38:44.976928 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971734 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:38:44.976928 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971739 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:38:44.976928 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971743 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:38:44.976928 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971746 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:38:44.976928 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971749 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:38:44.977405 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971751 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:38:44.977405 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971753 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:38:44.977405 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971756 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:38:44.977405 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971759 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:38:44.977405 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971761 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:38:44.977405 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971764 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:38:44.977405 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971766 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:38:44.977405 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971768 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:38:44.977405 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971772 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:38:44.977405 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971775 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:38:44.977405 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971777 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:38:44.977405 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971780 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:38:44.977405 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971782 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:38:44.977405 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971785 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:38:44.977405 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971787 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:38:44.977405 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971791 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:38:44.977405 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971793 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:38:44.977405 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971796 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:38:44.977405 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971799 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:38:44.977405 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971801 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:38:44.977930 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971803 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:38:44.977930 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971806 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:38:44.977930 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.971808 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:38:44.977930 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.971914 2579 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 20:38:44.977930 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.971922 2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 20:38:44.977930 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.971928 2579 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 20:38:44.977930 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.971933 2579 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 20:38:44.977930 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.971937 2579 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 20:38:44.977930 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.971940 2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 20:38:44.977930 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.971944 2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 20:38:44.977930 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.971949 2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 20:38:44.977930 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.971952 2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 20:38:44.977930 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.971955 2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 20:38:44.977930 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.971958 2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 20:38:44.977930 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.971961 2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 20:38:44.977930 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.971964 2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 20:38:44.977930 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.971967 2579 flags.go:64] FLAG: --cgroup-root="" Nov 03 20:38:44.977930 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.971970 2579 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 20:38:44.977930 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.971973 2579 flags.go:64] FLAG: --client-ca-file="" Nov 03 20:38:44.977930 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.971975 2579 flags.go:64] FLAG: --cloud-provider="external" Nov 03 20:38:44.977930 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.971978 2579 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 20:38:44.977930 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.971983 2579 flags.go:64] FLAG: --cluster-domain="" Nov 03 20:38:44.977930 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.971986 2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 20:38:44.977930 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.971990 2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 20:38:44.978604 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.971993 2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 20:38:44.978604 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.971997 2579 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 20:38:44.978604 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972003 2579 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 20:38:44.978604 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972008 2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 20:38:44.978604 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972013 2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 20:38:44.978604 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972018 2579 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 20:38:44.978604 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972022 2579 flags.go:64] FLAG: --contention-profiling="false" Nov 03 20:38:44.978604 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972028 2579 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 20:38:44.978604 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972032 2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 20:38:44.978604 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972037 2579 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 20:38:44.978604 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972042 2579 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 20:38:44.978604 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972047 2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 20:38:44.978604 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972050 2579 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 20:38:44.978604 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972053 2579 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 20:38:44.978604 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972056 2579 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 20:38:44.978604 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972059 2579 flags.go:64] FLAG: --enable-server="true" Nov 03 20:38:44.978604 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972063 2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 20:38:44.978604 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972067 2579 flags.go:64] FLAG: --event-burst="100" Nov 03 20:38:44.978604 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972070 2579 flags.go:64] FLAG: --event-qps="50" Nov 03 20:38:44.978604 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972073 2579 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 20:38:44.978604 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972076 2579 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 20:38:44.978604 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972078 2579 flags.go:64] FLAG: --eviction-hard="" Nov 03 20:38:44.978604 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972083 2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 20:38:44.978604 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972086 2579 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 20:38:44.979634 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972089 2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 20:38:44.979634 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972092 2579 flags.go:64] FLAG: --eviction-soft="" Nov 03 20:38:44.979634 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972095 2579 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 20:38:44.979634 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972098 2579 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 20:38:44.979634 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972101 2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 20:38:44.979634 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972104 2579 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 20:38:44.979634 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972107 2579 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 20:38:44.979634 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972110 2579 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 20:38:44.979634 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972113 2579 flags.go:64] FLAG: --feature-gates="" Nov 03 20:38:44.979634 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972117 2579 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 20:38:44.979634 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972120 2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 20:38:44.979634 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972123 2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 20:38:44.979634 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972126 2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 20:38:44.979634 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972129 2579 flags.go:64] FLAG: --healthz-port="10248" Nov 03 20:38:44.979634 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972132 2579 flags.go:64] FLAG: --help="false" Nov 03 20:38:44.979634 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972135 2579 flags.go:64] FLAG: --hostname-override="ip-10-0-128-175.us-west-1.compute.internal" Nov 03 20:38:44.979634 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972139 2579 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 20:38:44.979634 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972142 2579 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 20:38:44.979634 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972145 2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 20:38:44.979634 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972148 2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 20:38:44.979634 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972151 2579 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 20:38:44.979634 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972154 2579 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 20:38:44.979634 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972157 2579 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 20:38:44.980264 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972160 2579 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 20:38:44.980264 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972163 2579 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 20:38:44.980264 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972166 2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 20:38:44.980264 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972169 2579 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 20:38:44.980264 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972171 2579 flags.go:64] FLAG: --kube-reserved="" Nov 03 20:38:44.980264 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972174 2579 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 20:38:44.980264 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972177 2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 20:38:44.980264 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972180 2579 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 20:38:44.980264 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972183 2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 20:38:44.980264 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972185 2579 flags.go:64] FLAG: --lock-file="" Nov 03 20:38:44.980264 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972188 2579 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 20:38:44.980264 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972191 2579 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 20:38:44.980264 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972194 2579 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 20:38:44.980264 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972200 2579 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 20:38:44.980264 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972203 2579 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 20:38:44.980264 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972206 2579 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 20:38:44.980264 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972209 2579 flags.go:64] FLAG: --logging-format="text" Nov 03 20:38:44.980264 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972212 2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 20:38:44.980264 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972215 2579 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 20:38:44.980264 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972218 2579 flags.go:64] FLAG: --manifest-url="" Nov 03 20:38:44.980264 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972221 2579 flags.go:64] FLAG: --manifest-url-header="" Nov 03 20:38:44.980264 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972225 2579 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 20:38:44.980264 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972228 2579 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 20:38:44.980264 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972232 2579 flags.go:64] FLAG: --max-pods="110" Nov 03 20:38:44.980264 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972235 2579 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 20:38:44.980918 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972238 2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 20:38:44.980918 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972242 2579 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 20:38:44.980918 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972245 2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 20:38:44.980918 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972248 2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 20:38:44.980918 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972250 2579 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 20:38:44.980918 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972261 2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 20:38:44.980918 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972270 2579 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 20:38:44.980918 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972274 2579 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 20:38:44.980918 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972278 2579 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 20:38:44.980918 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972280 2579 flags.go:64] FLAG: --pod-cidr="" Nov 03 20:38:44.980918 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972283 2579 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 20:38:44.980918 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972286 2579 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 20:38:44.980918 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972289 2579 flags.go:64] FLAG: --pods-per-core="0" Nov 03 20:38:44.980918 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972292 2579 flags.go:64] FLAG: --port="10250" Nov 03 20:38:44.980918 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972295 2579 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 20:38:44.980918 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972298 2579 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-01cd9ab20b59b09db" Nov 03 20:38:44.980918 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972301 2579 flags.go:64] FLAG: --qos-reserved="" Nov 03 20:38:44.980918 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972304 2579 flags.go:64] FLAG: --read-only-port="10255" Nov 03 20:38:44.980918 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972307 2579 flags.go:64] FLAG: --register-node="true" Nov 03 20:38:44.980918 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972310 2579 flags.go:64] FLAG: --register-with-taints="" Nov 03 20:38:44.980918 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972315 2579 flags.go:64] FLAG: --registry-burst="10" Nov 03 20:38:44.980918 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972318 2579 flags.go:64] FLAG: --registry-qps="5" Nov 03 20:38:44.980918 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972321 2579 flags.go:64] FLAG: --reserved-cpus="" Nov 03 20:38:44.980918 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972325 2579 flags.go:64] FLAG: --reserved-memory="" Nov 03 20:38:44.980918 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972329 2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 20:38:44.980918 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972332 2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 20:38:44.981601 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972335 2579 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 20:38:44.981601 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972338 2579 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 20:38:44.981601 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972341 2579 flags.go:64] FLAG: --runonce="false" Nov 03 20:38:44.981601 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972344 2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 20:38:44.981601 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972347 2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 20:38:44.981601 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972350 2579 flags.go:64] FLAG: --seccomp-default="false" Nov 03 20:38:44.981601 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972353 2579 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 20:38:44.981601 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972356 2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 20:38:44.981601 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972360 2579 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 20:38:44.981601 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972364 2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 20:38:44.981601 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972367 2579 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 20:38:44.981601 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972370 2579 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 20:38:44.981601 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972373 2579 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 20:38:44.981601 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972377 2579 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 20:38:44.981601 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972380 2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 20:38:44.981601 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972383 2579 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 20:38:44.981601 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972386 2579 flags.go:64] FLAG: --system-cgroups="" Nov 03 20:38:44.981601 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972389 2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 20:38:44.981601 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972395 2579 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 20:38:44.981601 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972397 2579 flags.go:64] FLAG: --tls-cert-file="" Nov 03 20:38:44.981601 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972400 2579 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 20:38:44.981601 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972406 2579 flags.go:64] FLAG: --tls-min-version="" Nov 03 20:38:44.981601 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972408 2579 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 20:38:44.981601 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972411 2579 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 20:38:44.981601 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972414 2579 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 20:38:44.982265 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972417 2579 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 20:38:44.982265 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972422 2579 flags.go:64] FLAG: --v="2" Nov 03 20:38:44.982265 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972426 2579 flags.go:64] FLAG: --version="false" Nov 03 20:38:44.982265 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972445 2579 flags.go:64] FLAG: --vmodule="" Nov 03 20:38:44.982265 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972449 2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 20:38:44.982265 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972452 2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 20:38:44.982265 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972594 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:38:44.982265 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972598 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:38:44.982265 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972602 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:38:44.982265 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972605 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:38:44.982265 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972607 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:38:44.982265 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972610 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:38:44.982265 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972613 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:38:44.982265 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972615 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:38:44.982265 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972618 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:38:44.982265 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972620 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:38:44.982265 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972623 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:38:44.982265 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972626 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:38:44.982265 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972629 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:38:44.982265 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972631 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:38:44.982265 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972635 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:38:44.982838 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972638 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:38:44.982838 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972640 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:38:44.982838 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972643 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:38:44.982838 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972646 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:38:44.982838 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972648 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:38:44.982838 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972650 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:38:44.982838 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972653 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:38:44.982838 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972656 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:38:44.982838 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972659 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:38:44.982838 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972661 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:38:44.982838 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972664 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:38:44.982838 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972666 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:38:44.982838 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972670 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:38:44.982838 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972672 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:38:44.982838 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972675 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:38:44.982838 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972677 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:38:44.982838 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972680 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:38:44.982838 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972682 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:38:44.982838 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972685 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:38:44.982838 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972687 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:38:44.983355 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972690 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:38:44.983355 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972692 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:38:44.983355 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972695 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:38:44.983355 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972699 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:38:44.983355 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972703 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:38:44.983355 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972706 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:38:44.983355 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972709 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:38:44.983355 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972711 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:38:44.983355 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972714 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:38:44.983355 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972717 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:38:44.983355 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972719 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:38:44.983355 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972724 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:38:44.983355 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972728 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:38:44.983355 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972731 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:38:44.983355 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972734 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:38:44.983355 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972736 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:38:44.983355 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972739 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:38:44.983355 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972741 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:38:44.983866 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972744 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:38:44.983866 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972746 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:38:44.983866 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972749 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:38:44.983866 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972751 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:38:44.983866 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972753 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:38:44.983866 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972757 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:38:44.983866 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972760 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:38:44.983866 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972763 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:38:44.983866 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972766 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:38:44.983866 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972768 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:38:44.983866 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972770 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:38:44.983866 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972773 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:38:44.983866 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972776 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:38:44.983866 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972778 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:38:44.983866 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972781 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:38:44.983866 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972783 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:38:44.983866 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972786 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:38:44.983866 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972788 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:38:44.983866 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972790 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:38:44.984340 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972793 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:38:44.984340 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972795 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:38:44.984340 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972798 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:38:44.984340 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972800 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:38:44.984340 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972803 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:38:44.984340 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972806 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:38:44.984340 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972809 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:38:44.984340 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972812 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:38:44.984340 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972814 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:38:44.984340 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972817 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:38:44.984340 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972819 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:38:44.984340 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972822 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:38:44.984340 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972824 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:38:44.984340 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972827 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:38:44.984340 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972829 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:38:44.984340 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972832 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:38:44.984340 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972834 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:38:44.984340 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972837 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:38:44.984340 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972839 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:38:44.984340 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972843 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:38:44.984888 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972846 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:38:44.984888 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972848 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:38:44.984888 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972851 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:38:44.984888 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972853 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:38:44.984888 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972856 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:38:44.984888 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972858 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:38:44.984888 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972861 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:38:44.984888 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972864 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:38:44.984888 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972866 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:38:44.984888 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972869 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:38:44.984888 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972872 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:38:44.984888 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972874 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:38:44.984888 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972877 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:38:44.984888 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972879 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:38:44.984888 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.972906 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 20:38:44.984888 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972942 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:38:44.985301 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972947 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:38:44.985301 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972951 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:38:44.985301 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972954 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:38:44.985301 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972958 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:38:44.985301 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972961 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:38:44.985301 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972963 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:38:44.985301 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972966 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:38:44.985301 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972969 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:38:44.985301 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972971 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:38:44.985301 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972974 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:38:44.985301 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972976 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:38:44.985301 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972979 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:38:44.985301 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972981 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:38:44.985301 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972984 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:38:44.985301 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972986 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:38:44.985301 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972990 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:38:44.985301 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972993 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:38:44.985301 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972995 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:38:44.985301 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.972998 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:38:44.985301 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973001 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:38:44.985831 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973005 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:38:44.985831 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973008 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:38:44.985831 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973011 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:38:44.985831 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973015 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:38:44.985831 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973018 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:38:44.985831 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973020 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:38:44.985831 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973023 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:38:44.985831 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973026 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:38:44.985831 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973029 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:38:44.985831 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973033 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:38:44.985831 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973036 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:38:44.985831 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973038 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:38:44.985831 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973041 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:38:44.985831 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973043 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:38:44.985831 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973046 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:38:44.985831 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973050 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:38:44.985831 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973052 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:38:44.985831 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973055 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:38:44.986293 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973057 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:38:44.986293 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973060 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:38:44.986293 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973063 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:38:44.986293 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973066 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:38:44.986293 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973068 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:38:44.986293 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973071 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:38:44.986293 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973073 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:38:44.986293 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973075 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:38:44.986293 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973078 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:38:44.986293 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973082 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:38:44.986293 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973085 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:38:44.986293 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973087 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:38:44.986293 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973090 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:38:44.986293 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973092 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:38:44.986293 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973095 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:38:44.986293 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973097 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:38:44.986293 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973100 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:38:44.986293 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973102 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:38:44.986293 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973105 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:38:44.986797 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973107 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:38:44.986797 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973109 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:38:44.986797 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973112 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:38:44.986797 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973114 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:38:44.986797 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973117 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:38:44.986797 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973119 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:38:44.986797 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973122 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:38:44.986797 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973124 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:38:44.986797 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973127 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:38:44.986797 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973129 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:38:44.986797 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973135 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:38:44.986797 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973137 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:38:44.986797 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973140 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:38:44.986797 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973142 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:38:44.986797 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973145 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:38:44.986797 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973147 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:38:44.986797 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973149 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:38:44.986797 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973152 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:38:44.986797 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973155 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:38:44.986797 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973157 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:38:44.987309 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973159 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:38:44.987309 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973162 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:38:44.987309 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973166 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:38:44.987309 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973168 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:38:44.987309 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973170 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:38:44.987309 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973173 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:38:44.987309 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973176 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:38:44.987309 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973178 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:38:44.987309 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973180 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:38:44.987309 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973183 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:38:44.987309 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973185 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:38:44.987309 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973188 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:38:44.987309 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973191 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:38:44.987309 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973193 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:38:44.987309 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973196 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:38:44.987309 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973198 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:38:44.987309 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973200 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:38:44.987309 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973203 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:38:44.987309 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973205 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:38:44.987309 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973208 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:38:44.987834 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973210 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:38:44.987834 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973212 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:38:44.987834 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973216 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:38:44.987834 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973219 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:38:44.987834 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973223 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:38:44.987834 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973225 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:38:44.987834 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973228 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:38:44.987834 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.973230 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:38:44.987834 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.981220 2579 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 20:38:44.987834 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.981239 2579 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 20:38:44.987834 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.981257 2579 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 20:38:44.987834 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.981262 2579 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 20:38:44.987834 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981330 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:38:44.987834 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981335 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:38:44.987834 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981339 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:38:44.987834 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981342 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:38:44.987834 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981344 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:38:44.987834 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981347 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:38:44.987834 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981350 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:38:44.987834 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981353 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:38:44.988342 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981355 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:38:44.988342 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981358 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:38:44.988342 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981361 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:38:44.988342 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981363 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:38:44.988342 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981366 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:38:44.988342 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981368 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:38:44.988342 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981371 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:38:44.988342 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981374 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:38:44.988342 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981377 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:38:44.988342 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981380 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:38:44.988342 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981383 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:38:44.988342 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981385 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:38:44.988342 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981388 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:38:44.988342 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981391 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:38:44.988342 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981394 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:38:44.988342 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981396 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:38:44.988342 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981399 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:38:44.988342 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981402 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:38:44.988342 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981405 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:38:44.988342 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981407 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:38:44.988876 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981409 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:38:44.988876 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981413 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:38:44.988876 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981416 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:38:44.988876 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981419 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:38:44.988876 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981422 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:38:44.988876 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981426 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:38:44.988876 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981446 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:38:44.988876 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981450 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:38:44.988876 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981453 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:38:44.988876 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981455 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:38:44.988876 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981458 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:38:44.988876 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981461 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:38:44.988876 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981463 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:38:44.988876 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981466 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:38:44.988876 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981469 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:38:44.988876 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981471 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:38:44.988876 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981474 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:38:44.988876 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981476 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:38:44.988876 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981479 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:38:44.989354 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981482 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:38:44.989354 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981484 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:38:44.989354 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981487 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:38:44.989354 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981489 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:38:44.989354 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981492 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:38:44.989354 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981494 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:38:44.989354 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981498 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:38:44.989354 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981502 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:38:44.989354 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981505 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:38:44.989354 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981508 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:38:44.989354 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981511 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:38:44.989354 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981514 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:38:44.989354 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981516 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:38:44.989354 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981519 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:38:44.989354 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981522 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:38:44.989354 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981525 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:38:44.989354 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981527 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:38:44.989354 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981530 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:38:44.989354 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981532 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:38:44.989900 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981535 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:38:44.989900 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981537 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:38:44.989900 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981540 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:38:44.989900 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981542 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:38:44.989900 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981545 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:38:44.989900 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981547 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:38:44.989900 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981549 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:38:44.989900 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981552 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:38:44.989900 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981555 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:38:44.989900 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981557 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:38:44.989900 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981560 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:38:44.989900 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981562 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:38:44.989900 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981564 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:38:44.989900 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981568 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:38:44.989900 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981570 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:38:44.989900 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981573 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:38:44.989900 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981576 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:38:44.989900 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981578 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:38:44.989900 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981581 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:38:44.989900 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981583 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:38:44.990496 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981585 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:38:44.990496 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981588 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:38:44.990496 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981591 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:38:44.990496 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981593 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:38:44.990496 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981596 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:38:44.990496 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981598 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:38:44.990496 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981600 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:38:44.990496 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981603 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:38:44.990496 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981606 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:38:44.990496 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981609 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:38:44.990496 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981612 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:38:44.990496 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981615 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:38:44.990496 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981618 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:38:44.990496 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981621 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:38:44.990496 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981623 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:38:44.990496 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981626 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:38:44.990496 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981628 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:38:44.990496 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981631 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:38:44.990496 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981634 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:38:44.990496 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981636 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:38:44.991049 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.981667 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 20:38:44.991049 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981830 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:38:44.991049 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981835 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:38:44.991049 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981838 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:38:44.991049 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981840 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:38:44.991049 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981843 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:38:44.991049 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981847 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:38:44.991049 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981850 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:38:44.991049 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981853 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:38:44.991049 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981855 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:38:44.991049 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981860 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:38:44.991049 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981864 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:38:44.991049 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981867 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:38:44.991049 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981870 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:38:44.991049 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981873 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:38:44.991443 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981875 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:38:44.991443 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981878 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:38:44.991443 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981881 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:38:44.991443 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981883 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:38:44.991443 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981886 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:38:44.991443 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981888 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:38:44.991443 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981891 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:38:44.991443 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981893 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:38:44.991443 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981896 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:38:44.991443 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981898 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:38:44.991443 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981901 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:38:44.991443 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981904 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:38:44.991443 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981906 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:38:44.991443 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981909 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:38:44.991443 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981911 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:38:44.991443 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981914 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:38:44.991443 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981916 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:38:44.991443 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981918 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:38:44.991443 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981921 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:38:44.991443 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981923 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:38:44.991960 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981926 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:38:44.991960 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981928 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:38:44.991960 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981931 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:38:44.991960 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981933 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:38:44.991960 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981937 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:38:44.991960 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981940 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:38:44.991960 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981942 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:38:44.991960 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981945 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:38:44.991960 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981947 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:38:44.991960 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981950 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:38:44.991960 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981952 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:38:44.991960 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981955 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:38:44.991960 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981957 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:38:44.991960 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981959 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:38:44.991960 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981962 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:38:44.991960 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981964 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:38:44.991960 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981967 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:38:44.991960 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981969 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:38:44.991960 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981971 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:38:44.992455 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981974 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:38:44.992455 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981976 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:38:44.992455 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981979 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:38:44.992455 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981981 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:38:44.992455 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981984 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:38:44.992455 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981986 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:38:44.992455 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981989 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:38:44.992455 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981991 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:38:44.992455 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981994 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:38:44.992455 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981996 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:38:44.992455 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.981998 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:38:44.992455 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982001 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:38:44.992455 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982003 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:38:44.992455 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982006 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:38:44.992455 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982008 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:38:44.992455 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982011 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:38:44.992455 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982013 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:38:44.992455 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982016 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:38:44.992455 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982019 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:38:44.992455 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982022 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:38:44.992966 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982025 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:38:44.992966 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982029 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:38:44.992966 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982032 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:38:44.992966 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982035 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:38:44.992966 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982037 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:38:44.992966 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982040 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:38:44.992966 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982042 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:38:44.992966 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982045 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:38:44.992966 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982047 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:38:44.992966 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982050 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:38:44.992966 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982052 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:38:44.992966 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982055 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:38:44.992966 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982057 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:38:44.992966 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982060 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:38:44.992966 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982062 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:38:44.992966 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982064 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:38:44.992966 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982067 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:38:44.992966 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982070 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:38:44.992966 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982072 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:38:44.993600 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982075 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:38:44.993600 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982077 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:38:44.993600 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982080 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:38:44.993600 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982082 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:38:44.993600 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982085 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:38:44.993600 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982087 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:38:44.993600 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982089 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:38:44.993600 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982092 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:38:44.993600 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982094 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:38:44.993600 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982097 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:38:44.993600 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982099 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:38:44.993600 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982101 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:38:44.993600 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982108 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:38:44.993600 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982111 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:38:44.993600 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.982136 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 20:38:44.993600 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982168 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:38:44.994009 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982172 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:38:44.994009 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982174 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:38:44.994009 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982177 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:38:44.994009 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982180 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:38:44.994009 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982182 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:38:44.994009 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982185 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:38:44.994009 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982188 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:38:44.994009 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982192 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:38:44.994009 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982195 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:38:44.994009 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982197 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:38:44.994009 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982200 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:38:44.994009 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982202 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:38:44.994009 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982205 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:38:44.994009 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982207 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:38:44.994009 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982210 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:38:44.994009 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982213 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:38:44.994009 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982215 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:38:44.994009 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982218 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:38:44.994009 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982220 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:38:44.994512 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982223 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:38:44.994512 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982225 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:38:44.994512 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982228 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:38:44.994512 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982230 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:38:44.994512 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982233 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:38:44.994512 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982236 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:38:44.994512 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982238 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:38:44.994512 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982240 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:38:44.994512 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982243 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:38:44.994512 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982245 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:38:44.994512 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982248 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:38:44.994512 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982252 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:38:44.994512 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982255 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:38:44.994512 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982258 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:38:44.994512 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982260 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:38:44.994512 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982263 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:38:44.994512 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982265 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:38:44.994512 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982268 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:38:44.994512 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982270 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:38:44.994512 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982273 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:38:44.995009 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982276 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:38:44.995009 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982278 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:38:44.995009 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982281 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:38:44.995009 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982283 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:38:44.995009 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982285 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:38:44.995009 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982288 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:38:44.995009 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982290 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:38:44.995009 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982293 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:38:44.995009 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982295 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:38:44.995009 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982298 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:38:44.995009 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982300 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:38:44.995009 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982303 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:38:44.995009 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982305 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:38:44.995009 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982308 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:38:44.995009 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982310 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:38:44.995009 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982313 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:38:44.995009 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982315 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:38:44.995009 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982318 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:38:44.995009 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982320 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:38:44.995009 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982323 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:38:44.995532 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982325 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:38:44.995532 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982328 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:38:44.995532 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982330 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:38:44.995532 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982334 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:38:44.995532 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982337 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:38:44.995532 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982339 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:38:44.995532 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982342 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:38:44.995532 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982344 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:38:44.995532 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982346 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:38:44.995532 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982350 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:38:44.995532 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982352 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:38:44.995532 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982356 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:38:44.995532 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982359 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:38:44.995532 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982362 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:38:44.995532 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982365 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:38:44.995532 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982376 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:38:44.995532 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982380 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:38:44.995532 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982382 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:38:44.995532 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982385 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:38:44.996007 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982389 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:38:44.996007 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982393 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:38:44.996007 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982396 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:38:44.996007 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982399 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:38:44.996007 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982401 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:38:44.996007 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982404 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:38:44.996007 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982406 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:38:44.996007 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982409 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:38:44.996007 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982411 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:38:44.996007 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982415 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:38:44.996007 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982418 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:38:44.996007 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982421 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:38:44.996007 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982423 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:38:44.996007 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982426 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:38:44.996007 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982445 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:38:44.996007 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982450 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:38:44.996007 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982453 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:38:44.996007 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982456 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:38:44.996007 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982459 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:38:44.996618 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982462 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:38:44.996618 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982465 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:38:44.996618 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982467 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:38:44.996618 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982470 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:38:44.996618 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982472 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:38:44.996618 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982475 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:38:44.996618 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982477 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:38:44.996618 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:44.982480 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:38:44.996618 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.982563 2579 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 20:38:44.996618 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.984590 2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 03 20:38:44.996618 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.985460 2579 server.go:1014] "Starting client certificate rotation" Nov 03 20:38:44.996618 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.985553 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 20:38:44.996618 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.985590 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 20:38:44.996618 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.987552 2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 20:38:44.996618 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.989579 2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 20:38:44.999840 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:44.999818 2579 log.go:25] "Validated CRI v1 runtime API" Nov 03 20:38:45.003491 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.003472 2579 log.go:25] "Validated CRI v1 image API" Nov 03 20:38:45.004342 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.004314 2579 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 20:38:45.007017 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.006995 2579 fs.go:135] Filesystem UUIDs: map[052e3f40-4b72-478f-bc3f-d4be5679e7e8:/dev/nvme0n1p3 69cfa50c-3477-4765-b594-c223f3a188f4:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Nov 03 20:38:45.007085 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.007016 2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 03 20:38:45.013310 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.013172 2579 manager.go:217] Machine: {Timestamp:2026-11-03 20:38:45.011867398 +0000 UTC m=+0.316202458 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100473 MemoryCapacity:32812007424 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e10b2ff5b3ea62e589b49f2e14a92 SystemUUID:ec2e10b2-ff5b-3ea6-2e58-9b49f2e14a92 BootID:6840dd11-ad1f-4708-8cfc-1d96c63aaed5 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406003712 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:15:22:95:81:11 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:15:22:95:81:11 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:96:61:b7:b5:b6:c1 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812007424 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 20:38:45.014043 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.014030 2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 20:38:45.014198 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.014182 2579 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 20:38:45.016674 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.016647 2579 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 20:38:45.016813 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.016675 2579 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-175.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 20:38:45.016868 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.016815 2579 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 20:38:45.016868 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.016823 2579 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 20:38:45.016868 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.016840 2579 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 20:38:45.017752 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.017740 2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 20:38:45.018510 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.018499 2579 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 20:38:45.023259 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.023237 2579 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 20:38:45.023381 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.023372 2579 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 20:38:45.023946 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.023936 2579 kubelet.go:493] "Attempting to sync node with API server" Nov 03 20:38:45.023988 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.023950 2579 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 20:38:45.023988 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.023961 2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 20:38:45.023988 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.023970 2579 kubelet.go:405] "Adding apiserver pod source" Nov 03 20:38:45.023988 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.023978 2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 20:38:45.024420 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.024399 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 20:38:45.026663 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.026641 2579 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 20:38:45.031692 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.031675 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 20:38:45.032401 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.032389 2579 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 20:38:45.032458 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.032421 2579 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 20:38:45.032603 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.032593 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 20:38:45.032635 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.032610 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 20:38:45.032635 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.032617 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 20:38:45.032635 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.032623 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 20:38:45.032635 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.032628 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 20:38:45.032749 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.032638 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 20:38:45.033309 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.033300 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 20:38:45.033344 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.033313 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 20:38:45.033344 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.033320 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 20:38:45.033344 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.033326 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 20:38:45.033344 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.033335 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 20:38:45.033344 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.033345 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 20:38:45.034059 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.034049 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 20:38:45.034094 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.034063 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 20:38:45.047660 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.047642 2579 server.go:1263] "Started kubelet" Nov 03 20:38:45.047769 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.047724 2579 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 20:38:45.047822 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.047749 2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 20:38:45.047877 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.047830 2579 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 20:38:45.048334 ip-10-0-128-175 systemd[1]: Started Kubernetes Kubelet. Nov 03 20:38:45.049181 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.049162 2579 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 20:38:45.050626 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.050610 2579 server.go:317] "Adding debug handlers to kubelet server" Nov 03 20:38:45.053504 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:45.053483 2579 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 03 20:38:45.053989 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.053973 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 20:38:45.054509 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.054491 2579 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 20:38:45.054689 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:45.054667 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-175.us-west-1.compute.internal\" not found" Nov 03 20:38:45.055289 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.055272 2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 20:38:45.055289 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.055275 2579 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 20:38:45.055289 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.055295 2579 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 20:38:45.055487 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.055317 2579 factory.go:55] Registering systemd factory Nov 03 20:38:45.055487 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.055331 2579 factory.go:223] Registration of the systemd container factory successfully Nov 03 20:38:45.055487 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.055403 2579 reconstruct.go:97] "Volume reconstruction finished" Nov 03 20:38:45.055624 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.055497 2579 reconciler.go:29] "Reconciler: start to sync state" Nov 03 20:38:45.055624 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.055579 2579 factory.go:153] Registering CRI-O factory Nov 03 20:38:45.055624 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.055594 2579 factory.go:223] Registration of the crio container factory successfully Nov 03 20:38:45.055753 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.055643 2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 20:38:45.055753 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.055661 2579 factory.go:103] Registering Raw factory Nov 03 20:38:45.055753 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.055675 2579 manager.go:1199] Started watching for new ooms in manager Nov 03 20:38:45.056549 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.056534 2579 manager.go:319] Starting recovery of all containers Nov 03 20:38:45.069095 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.069063 2579 manager.go:324] Recovery completed Nov 03 20:38:45.073215 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.073195 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:38:45.074952 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.074926 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-175.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:38:45.075024 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.074973 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-175.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:38:45.075024 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.074985 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-175.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 20:38:45.075538 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.075522 2579 cpu_manager.go:226] "Starting" policy="none" Nov 03 20:38:45.075538 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.075535 2579 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 20:38:45.075650 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.075552 2579 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 20:38:45.075650 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.075572 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:38:45.076232 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.076217 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-175.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:38:45.076301 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.076251 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-175.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:38:45.076301 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.076266 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-175.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 20:38:45.077915 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.077898 2579 policy_none.go:50] "Start" Nov 03 20:38:45.077982 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.077923 2579 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 20:38:45.077982 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.077942 2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 20:38:45.080049 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.080037 2579 policy_none.go:44] "Start" Nov 03 20:38:45.119240 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.119042 2579 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 20:38:45.123568 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.123554 2579 manager.go:341] "Starting Device Plugin manager" Nov 03 20:38:45.123623 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:45.123585 2579 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 20:38:45.123623 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.123595 2579 server.go:87] "Starting device plugin registration server" Nov 03 20:38:45.123911 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.123899 2579 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 20:38:45.123944 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.123915 2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 20:38:45.124032 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.124016 2579 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 20:38:45.124134 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.124116 2579 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 20:38:45.124134 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.124130 2579 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 20:38:45.124622 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:45.124605 2579 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 20:38:45.124678 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:45.124660 2579 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-175.us-west-1.compute.internal\" not found" Nov 03 20:38:45.178833 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.178793 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 20:38:45.180739 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.180087 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 20:38:45.180739 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.180106 2579 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 20:38:45.180739 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.180129 2579 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 20:38:45.180739 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:45.180160 2579 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 03 20:38:45.190683 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:45.190654 2579 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-175.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 03 20:38:45.190683 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.190664 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-175.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 03 20:38:45.191712 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:45.190667 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-175.us-west-1.compute.internal.18e4a380c4775bea default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-175.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-175.us-west-1.compute.internal,},FirstTimestamp:2026-11-03 20:38:45.047598058 +0000 UTC m=+0.351933118,LastTimestamp:2026-11-03 20:38:45.047598058 +0000 UTC m=+0.351933118,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-175.us-west-1.compute.internal,}" Nov 03 20:38:45.224455 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.224381 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:38:45.225281 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.225264 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-175.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:38:45.225363 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.225294 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-175.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:38:45.225363 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.225305 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-175.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 20:38:45.225363 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.225333 2579 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-175.us-west-1.compute.internal" Nov 03 20:38:45.271655 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.271620 2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 20:38:45.271805 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.271687 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-shkx7" Nov 03 20:38:45.280864 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.280840 2579 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-175.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-175.us-west-1.compute.internal"] Nov 03 20:38:45.280943 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.280903 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:38:45.281816 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.281796 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-175.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:38:45.281872 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.281838 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-175.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:38:45.281872 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.281852 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-175.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 20:38:45.284028 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.284016 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:38:45.284164 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.284149 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-175.us-west-1.compute.internal" Nov 03 20:38:45.284204 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.284180 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:38:45.284743 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.284720 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-175.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:38:45.284743 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.284747 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-175.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:38:45.284880 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.284756 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-175.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 20:38:45.284880 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.284722 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-175.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:38:45.284880 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.284811 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-175.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:38:45.284880 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.284825 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-175.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 20:38:45.286740 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.286727 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-175.us-west-1.compute.internal" Nov 03 20:38:45.286802 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.286752 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:38:45.287468 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.287451 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-175.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:38:45.287554 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.287480 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-175.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:38:45.287554 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.287490 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-175.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 20:38:45.304856 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:45.304838 2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-175.us-west-1.compute.internal\" not found" node="ip-10-0-128-175.us-west-1.compute.internal" Nov 03 20:38:45.306892 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:45.306878 2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-175.us-west-1.compute.internal\" not found" node="ip-10-0-128-175.us-west-1.compute.internal" Nov 03 20:38:45.351298 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.351269 2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:38:45.356731 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.356706 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c6e5ca9eba1ab368215b5511cd840a25-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-175.us-west-1.compute.internal\" (UID: \"c6e5ca9eba1ab368215b5511cd840a25\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-175.us-west-1.compute.internal" Nov 03 20:38:45.356813 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.356734 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6e5ca9eba1ab368215b5511cd840a25-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-175.us-west-1.compute.internal\" (UID: \"c6e5ca9eba1ab368215b5511cd840a25\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-175.us-west-1.compute.internal" Nov 03 20:38:45.356813 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.356751 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6873aa06186bc61b65aa0fa7f8c6d9a2-config\") pod \"kube-apiserver-proxy-ip-10-0-128-175.us-west-1.compute.internal\" (UID: \"6873aa06186bc61b65aa0fa7f8c6d9a2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-175.us-west-1.compute.internal" Nov 03 20:38:45.402797 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.402770 2579 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:38:45.450724 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.450700 2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:38:45.452812 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.452783 2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:38:45.454217 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.453409 2579 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-175.us-west-1.compute.internal" Nov 03 20:38:45.454217 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:45.453462 2579 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-175.us-west-1.compute.internal\": node \"ip-10-0-128-175.us-west-1.compute.internal\" not found" Nov 03 20:38:45.457816 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.457795 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6e5ca9eba1ab368215b5511cd840a25-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-175.us-west-1.compute.internal\" (UID: \"c6e5ca9eba1ab368215b5511cd840a25\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-175.us-west-1.compute.internal" Nov 03 20:38:45.457878 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.457825 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6873aa06186bc61b65aa0fa7f8c6d9a2-config\") pod \"kube-apiserver-proxy-ip-10-0-128-175.us-west-1.compute.internal\" (UID: \"6873aa06186bc61b65aa0fa7f8c6d9a2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-175.us-west-1.compute.internal" Nov 03 20:38:45.457878 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.457841 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c6e5ca9eba1ab368215b5511cd840a25-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-175.us-west-1.compute.internal\" (UID: \"c6e5ca9eba1ab368215b5511cd840a25\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-175.us-west-1.compute.internal" Nov 03 20:38:45.457941 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.457895 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c6e5ca9eba1ab368215b5511cd840a25-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-175.us-west-1.compute.internal\" (UID: \"c6e5ca9eba1ab368215b5511cd840a25\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-175.us-west-1.compute.internal" Nov 03 20:38:45.457941 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.457909 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6873aa06186bc61b65aa0fa7f8c6d9a2-config\") pod \"kube-apiserver-proxy-ip-10-0-128-175.us-west-1.compute.internal\" (UID: \"6873aa06186bc61b65aa0fa7f8c6d9a2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-175.us-west-1.compute.internal" Nov 03 20:38:45.457941 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.457906 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6e5ca9eba1ab368215b5511cd840a25-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-175.us-west-1.compute.internal\" (UID: \"c6e5ca9eba1ab368215b5511cd840a25\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-175.us-west-1.compute.internal" Nov 03 20:38:45.555622 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.555529 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-175.us-west-1.compute.internal" Nov 03 20:38:45.608748 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.608716 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-175.us-west-1.compute.internal" Nov 03 20:38:45.609828 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.609794 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-175.us-west-1.compute.internal" Nov 03 20:38:45.626338 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.626319 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 20:38:45.626458 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.626446 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-175.us-west-1.compute.internal" Nov 03 20:38:45.682830 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.682809 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 20:38:45.985887 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.985803 2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 03 20:38:45.986487 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.985966 2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 20:38:45.986487 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.985996 2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 20:38:45.986487 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:45.986011 2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 20:38:45.986487 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:45.985990 2579 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a0165055bbafc464b85f5db0d8234891-51f743d41745d646.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-128-175.us-west-1.compute.internal.18e4a380c6194e76\": read tcp 10.0.128.175:49256->3.21.224.121:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-128-175.us-west-1.compute.internal.18e4a380c6194e76 default 5165 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-175.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-128-175.us-west-1.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-128-175.us-west-1.compute.internal,},FirstTimestamp:2026-11-03 20:38:45 +0000 UTC,LastTimestamp:2026-11-03 20:38:45.225309181 +0000 UTC m=+0.529644241,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-175.us-west-1.compute.internal,}" Nov 03 20:38:46.024159 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.024128 2579 apiserver.go:52] "Watching apiserver" Nov 03 20:38:46.054530 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.054498 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 03 20:38:46.276271 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.276159 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 20:33:46 +0000 UTC" deadline="2028-07-16 06:00:49.909689944 +0000 UTC" Nov 03 20:38:46.276271 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.276213 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14889h22m3.633479925s" Nov 03 20:38:46.304013 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.303986 2579 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 20:38:46.304343 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.304327 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-6jqz7","openshift-multus/network-metrics-daemon-xt5qx","kube-system/konnectivity-agent-jvtt8","openshift-multus/multus-4k6r2","openshift-network-diagnostics/network-check-target-hc89p","openshift-ovn-kubernetes/ovnkube-node-4ljxs","openshift-dns/node-resolver-gkm7b","openshift-image-registry/node-ca-p6hwx","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-175.us-west-1.compute.internal","openshift-multus/multus-additional-cni-plugins-shzwn","openshift-network-operator/iptables-alerter-lvcbj","kube-system/global-pull-secret-syncer-g4892","kube-system/kube-apiserver-proxy-ip-10-0-128-175.us-west-1.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq7n8"] Nov 03 20:38:46.306942 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.306921 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:38:46.308936 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.308915 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xt5qx" Nov 03 20:38:46.309027 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:46.308996 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xt5qx" podUID="747345eb-a407-423b-92df-c5d765a7da2d" Nov 03 20:38:46.310946 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.310932 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jvtt8" Nov 03 20:38:46.312978 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.312965 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4k6r2" Nov 03 20:38:46.315088 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.315073 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hc89p" Nov 03 20:38:46.315149 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:46.315135 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hc89p" podUID="2dfb2512-b394-4a37-a5ba-a783d4e46c46" Nov 03 20:38:46.317266 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.317239 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:38:46.319494 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.319474 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gkm7b" Nov 03 20:38:46.321647 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.321633 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-p6hwx" Nov 03 20:38:46.323901 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.323888 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-shzwn" Nov 03 20:38:46.326043 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.326030 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lvcbj" Nov 03 20:38:46.328218 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.328200 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g4892" Nov 03 20:38:46.328287 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:46.328269 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g4892" podUID="03985fe9-0dfe-4f9a-97db-bb4c0298be5c" Nov 03 20:38:46.330542 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.330528 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq7n8" Nov 03 20:38:46.356328 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.356301 2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 20:38:46.362213 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.362184 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 20:38:46.362333 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.362249 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-r6k78\"" Nov 03 20:38:46.362333 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.362273 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:38:46.363423 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.363403 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f1f5ba6-ccc5-4f0d-a135-743747ce667a-cnibin\") pod \"multus-additional-cni-plugins-shzwn\" (UID: \"2f1f5ba6-ccc5-4f0d-a135-743747ce667a\") " pod="openshift-multus/multus-additional-cni-plugins-shzwn" Nov 03 20:38:46.363499 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.363455 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-host\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:38:46.363499 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.363480 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-etc-tuned\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:38:46.363574 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.363541 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-host-var-lib-cni-bin\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:38:46.363620 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.363580 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-run-systemd\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:38:46.363620 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.363603 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-log-socket\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:38:46.363699 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.363626 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/12658495-b5af-4c01-9c72-19b1173f7529-ovnkube-script-lib\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:38:46.363699 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.363667 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6e89ebac-2191-4acc-b4d4-ff326b3b74ac-iptables-alerter-script\") pod \"iptables-alerter-lvcbj\" (UID: \"6e89ebac-2191-4acc-b4d4-ff326b3b74ac\") " pod="openshift-network-operator/iptables-alerter-lvcbj" Nov 03 20:38:46.363788 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.363722 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-multus-socket-dir-parent\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:38:46.363838 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.363790 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25sxl\" (UniqueName: \"kubernetes.io/projected/bafd96d7-6982-4c93-81fc-d61643002fd1-kube-api-access-25sxl\") pod \"aws-ebs-csi-driver-node-jq7n8\" (UID: \"bafd96d7-6982-4c93-81fc-d61643002fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq7n8" Nov 03 20:38:46.363838 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.363820 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-systemd-units\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:38:46.363993 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.363844 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/12658495-b5af-4c01-9c72-19b1173f7529-ovn-node-metrics-cert\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:38:46.363993 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.363865 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z294f\" (UniqueName: \"kubernetes.io/projected/2f1f5ba6-ccc5-4f0d-a135-743747ce667a-kube-api-access-z294f\") pod \"multus-additional-cni-plugins-shzwn\" (UID: \"2f1f5ba6-ccc5-4f0d-a135-743747ce667a\") " pod="openshift-multus/multus-additional-cni-plugins-shzwn" Nov 03 20:38:46.363993 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.363899 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-host-run-multus-certs\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:38:46.363993 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.363956 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bafd96d7-6982-4c93-81fc-d61643002fd1-registration-dir\") pod \"aws-ebs-csi-driver-node-jq7n8\" (UID: \"bafd96d7-6982-4c93-81fc-d61643002fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq7n8" Nov 03 20:38:46.363993 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.363985 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-host-cni-bin\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:38:46.364169 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.364005 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-etc-sysconfig\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:38:46.364169 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.364051 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-lib-modules\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:38:46.364169 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.364095 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-hostroot\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:38:46.364169 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.364112 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4a5df9f4-aaaa-4441-a2e0-28744e197828-multus-daemon-config\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:38:46.364169 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.364127 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2vcp\" (UniqueName: \"kubernetes.io/projected/4a5df9f4-aaaa-4441-a2e0-28744e197828-kube-api-access-z2vcp\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:38:46.364169 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.364140 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-host-kubelet\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:38:46.364169 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.364157 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:38:46.364169 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.364172 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cf15bf7c-07c6-4d82-987b-aa4a332d5ca9-hosts-file\") pod \"node-resolver-gkm7b\" (UID: \"cf15bf7c-07c6-4d82-987b-aa4a332d5ca9\") " pod="openshift-dns/node-resolver-gkm7b" Nov 03 20:38:46.364539 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.364197 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8b6m\" (UniqueName: \"kubernetes.io/projected/cf15bf7c-07c6-4d82-987b-aa4a332d5ca9-kube-api-access-t8b6m\") pod \"node-resolver-gkm7b\" (UID: \"cf15bf7c-07c6-4d82-987b-aa4a332d5ca9\") " pod="openshift-dns/node-resolver-gkm7b" Nov 03 20:38:46.364539 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.364252 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-etc-sysctl-d\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:38:46.364539 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.364282 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-etc-sysctl-conf\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:38:46.364539 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.364306 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-tmp\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:38:46.364539 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.364328 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7ced550a-b3cb-4605-b8dc-aab3b55804b4-host\") pod \"node-ca-p6hwx\" (UID: \"7ced550a-b3cb-4605-b8dc-aab3b55804b4\") " pod="openshift-image-registry/node-ca-p6hwx" Nov 03 20:38:46.364539 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.364350 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-var-lib-kubelet\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:38:46.364539 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.364373 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3e72f317-1042-40b0-b709-dc3d650754b5-agent-certs\") pod \"konnectivity-agent-jvtt8\" (UID: \"3e72f317-1042-40b0-b709-dc3d650754b5\") " pod="kube-system/konnectivity-agent-jvtt8" Nov 03 20:38:46.364539 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.364398 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3e72f317-1042-40b0-b709-dc3d650754b5-konnectivity-ca\") pod \"konnectivity-agent-jvtt8\" (UID: \"3e72f317-1042-40b0-b709-dc3d650754b5\") " pod="kube-system/konnectivity-agent-jvtt8" Nov 03 20:38:46.364539 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.364444 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f1f5ba6-ccc5-4f0d-a135-743747ce667a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-shzwn\" (UID: \"2f1f5ba6-ccc5-4f0d-a135-743747ce667a\") " pod="openshift-multus/multus-additional-cni-plugins-shzwn" Nov 03 20:38:46.364539 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.364464 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6e89ebac-2191-4acc-b4d4-ff326b3b74ac-host-slash\") pod \"iptables-alerter-lvcbj\" (UID: \"6e89ebac-2191-4acc-b4d4-ff326b3b74ac\") " pod="openshift-network-operator/iptables-alerter-lvcbj" Nov 03 20:38:46.364539 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.364478 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-var-lib-openvswitch\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:38:46.364539 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.364493 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-etc-openvswitch\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:38:46.364539 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.364514 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-run-ovn\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:38:46.364539 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.364528 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmbq4\" (UniqueName: \"kubernetes.io/projected/6e89ebac-2191-4acc-b4d4-ff326b3b74ac-kube-api-access-tmbq4\") pod \"iptables-alerter-lvcbj\" (UID: \"6e89ebac-2191-4acc-b4d4-ff326b3b74ac\") " pod="openshift-network-operator/iptables-alerter-lvcbj" Nov 03 20:38:46.365025 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.364541 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-host-run-netns\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:38:46.365025 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.364562 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-multus-conf-dir\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:38:46.365025 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.364582 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-run-openvswitch\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:38:46.365025 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.364609 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf15bf7c-07c6-4d82-987b-aa4a332d5ca9-tmp-dir\") pod \"node-resolver-gkm7b\" (UID: \"cf15bf7c-07c6-4d82-987b-aa4a332d5ca9\") " pod="openshift-dns/node-resolver-gkm7b" Nov 03 20:38:46.365025 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.364623 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/03985fe9-0dfe-4f9a-97db-bb4c0298be5c-dbus\") pod \"global-pull-secret-syncer-g4892\" (UID: \"03985fe9-0dfe-4f9a-97db-bb4c0298be5c\") " pod="kube-system/global-pull-secret-syncer-g4892" Nov 03 20:38:46.365025 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.364637 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-cnibin\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:38:46.365025 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.364652 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-host-var-lib-cni-multus\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:38:46.365025 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.364675 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bafd96d7-6982-4c93-81fc-d61643002fd1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jq7n8\" (UID: \"bafd96d7-6982-4c93-81fc-d61643002fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq7n8" Nov 03 20:38:46.365025 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.364717 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/12658495-b5af-4c01-9c72-19b1173f7529-env-overrides\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:38:46.365025 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.364778 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-sys\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:38:46.365025 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.364807 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj57q\" (UniqueName: \"kubernetes.io/projected/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-kube-api-access-wj57q\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:38:46.365025 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.364828 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bafd96d7-6982-4c93-81fc-d61643002fd1-sys-fs\") pod \"aws-ebs-csi-driver-node-jq7n8\" (UID: \"bafd96d7-6982-4c93-81fc-d61643002fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq7n8" Nov 03 20:38:46.365025 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.364848 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-node-log\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:38:46.365025 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.364865 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/12658495-b5af-4c01-9c72-19b1173f7529-ovnkube-config\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:38:46.365025 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.364881 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03985fe9-0dfe-4f9a-97db-bb4c0298be5c-original-pull-secret\") pod \"global-pull-secret-syncer-g4892\" (UID: \"03985fe9-0dfe-4f9a-97db-bb4c0298be5c\") " pod="kube-system/global-pull-secret-syncer-g4892" Nov 03 20:38:46.365025 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.364931 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-etc-kubernetes\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:38:46.365025 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.364977 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-run\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:38:46.365740 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.365002 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-system-cni-dir\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:38:46.365740 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.365025 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bafd96d7-6982-4c93-81fc-d61643002fd1-socket-dir\") pod \"aws-ebs-csi-driver-node-jq7n8\" (UID: \"bafd96d7-6982-4c93-81fc-d61643002fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq7n8" Nov 03 20:38:46.365740 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.365050 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-host-run-netns\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:38:46.365740 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.365075 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cscp\" (UniqueName: \"kubernetes.io/projected/12658495-b5af-4c01-9c72-19b1173f7529-kube-api-access-6cscp\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:38:46.365740 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.365102 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2f1f5ba6-ccc5-4f0d-a135-743747ce667a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-shzwn\" (UID: \"2f1f5ba6-ccc5-4f0d-a135-743747ce667a\") " pod="openshift-multus/multus-additional-cni-plugins-shzwn" Nov 03 20:38:46.365740 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.365138 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-etc-systemd\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:38:46.365740 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.365166 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f1f5ba6-ccc5-4f0d-a135-743747ce667a-system-cni-dir\") pod \"multus-additional-cni-plugins-shzwn\" (UID: \"2f1f5ba6-ccc5-4f0d-a135-743747ce667a\") " pod="openshift-multus/multus-additional-cni-plugins-shzwn" Nov 03 20:38:46.365740 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.365190 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f1f5ba6-ccc5-4f0d-a135-743747ce667a-cni-binary-copy\") pod \"multus-additional-cni-plugins-shzwn\" (UID: \"2f1f5ba6-ccc5-4f0d-a135-743747ce667a\") " pod="openshift-multus/multus-additional-cni-plugins-shzwn" Nov 03 20:38:46.365740 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.365214 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4a5df9f4-aaaa-4441-a2e0-28744e197828-cni-binary-copy\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:38:46.365740 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.365237 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6rjq\" (UniqueName: \"kubernetes.io/projected/7ced550a-b3cb-4605-b8dc-aab3b55804b4-kube-api-access-t6rjq\") pod \"node-ca-p6hwx\" (UID: \"7ced550a-b3cb-4605-b8dc-aab3b55804b4\") " pod="openshift-image-registry/node-ca-p6hwx" Nov 03 20:38:46.365740 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.365260 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/747345eb-a407-423b-92df-c5d765a7da2d-metrics-certs\") pod \"network-metrics-daemon-xt5qx\" (UID: \"747345eb-a407-423b-92df-c5d765a7da2d\") " pod="openshift-multus/network-metrics-daemon-xt5qx" Nov 03 20:38:46.365740 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.365287 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2brl\" (UniqueName: \"kubernetes.io/projected/747345eb-a407-423b-92df-c5d765a7da2d-kube-api-access-l2brl\") pod \"network-metrics-daemon-xt5qx\" (UID: \"747345eb-a407-423b-92df-c5d765a7da2d\") " pod="openshift-multus/network-metrics-daemon-xt5qx" Nov 03 20:38:46.365740 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.365313 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-host-run-ovn-kubernetes\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:38:46.365740 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.365345 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-etc-modprobe-d\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:38:46.365740 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.365367 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-host-var-lib-kubelet\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:38:46.365740 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.365391 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-etc-kubernetes\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:38:46.366220 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.365444 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2f1f5ba6-ccc5-4f0d-a135-743747ce667a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-shzwn\" (UID: \"2f1f5ba6-ccc5-4f0d-a135-743747ce667a\") " pod="openshift-multus/multus-additional-cni-plugins-shzwn" Nov 03 20:38:46.366220 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.365471 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-multus-cni-dir\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:38:46.366220 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.365494 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7ced550a-b3cb-4605-b8dc-aab3b55804b4-serviceca\") pod \"node-ca-p6hwx\" (UID: \"7ced550a-b3cb-4605-b8dc-aab3b55804b4\") " pod="openshift-image-registry/node-ca-p6hwx" Nov 03 20:38:46.366220 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.365541 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bafd96d7-6982-4c93-81fc-d61643002fd1-etc-selinux\") pod \"aws-ebs-csi-driver-node-jq7n8\" (UID: \"bafd96d7-6982-4c93-81fc-d61643002fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq7n8" Nov 03 20:38:46.366220 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.365587 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rlc8\" (UniqueName: \"kubernetes.io/projected/2dfb2512-b394-4a37-a5ba-a783d4e46c46-kube-api-access-5rlc8\") pod \"network-check-target-hc89p\" (UID: \"2dfb2512-b394-4a37-a5ba-a783d4e46c46\") " pod="openshift-network-diagnostics/network-check-target-hc89p" Nov 03 20:38:46.366220 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.365614 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-host-slash\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:38:46.366220 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.365638 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-host-cni-netd\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:38:46.366220 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.365666 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/03985fe9-0dfe-4f9a-97db-bb4c0298be5c-kubelet-config\") pod \"global-pull-secret-syncer-g4892\" (UID: \"03985fe9-0dfe-4f9a-97db-bb4c0298be5c\") " pod="kube-system/global-pull-secret-syncer-g4892" Nov 03 20:38:46.366220 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.365690 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-host-run-k8s-cni-cncf-io\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:38:46.366220 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.365712 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bafd96d7-6982-4c93-81fc-d61643002fd1-device-dir\") pod \"aws-ebs-csi-driver-node-jq7n8\" (UID: \"bafd96d7-6982-4c93-81fc-d61643002fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq7n8" Nov 03 20:38:46.366692 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.366556 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-kqp9s\"" Nov 03 20:38:46.366692 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.366604 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 20:38:46.366692 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.366673 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-29dtl\"" Nov 03 20:38:46.366942 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.366924 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 20:38:46.366997 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.366950 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 20:38:46.366997 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.366989 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 20:38:46.366997 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.366994 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 20:38:46.367138 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.367054 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 20:38:46.371229 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.371210 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 20:38:46.371332 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.371287 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-nnzhp\"" Nov 03 20:38:46.371695 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.371682 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 20:38:46.371753 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.371693 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 20:38:46.371753 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.371714 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 20:38:46.416040 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.416016 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4xfms\"" Nov 03 20:38:46.458279 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.458246 2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 20:38:46.458468 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.458290 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-btx2l" Nov 03 20:38:46.462783 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.462757 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 20:38:46.462931 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.462803 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 20:38:46.462931 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.462819 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 20:38:46.462931 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.462828 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 20:38:46.463099 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.463083 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 20:38:46.463187 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.463131 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 20:38:46.463187 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.463140 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 20:38:46.463187 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.463143 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-czwwq\"" Nov 03 20:38:46.463187 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.463147 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 20:38:46.463187 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.463138 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 20:38:46.463187 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.463187 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 20:38:46.463470 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.463203 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 20:38:46.463470 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.463206 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dm4wl\"" Nov 03 20:38:46.463470 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.463219 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 20:38:46.463470 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.463281 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zxrmv\"" Nov 03 20:38:46.463470 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.463387 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:38:46.463470 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.463421 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8nf42\"" Nov 03 20:38:46.463470 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.463462 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 20:38:46.463663 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.463577 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 20:38:46.466777 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.466762 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-etc-modprobe-d\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:38:46.466817 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.466785 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-host-var-lib-kubelet\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:38:46.466817 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.466799 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-etc-kubernetes\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:38:46.466888 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.466818 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2f1f5ba6-ccc5-4f0d-a135-743747ce667a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-shzwn\" (UID: \"2f1f5ba6-ccc5-4f0d-a135-743747ce667a\") " pod="openshift-multus/multus-additional-cni-plugins-shzwn" Nov 03 20:38:46.466888 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.466832 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-multus-cni-dir\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:38:46.466993 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.466891 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-etc-kubernetes\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:38:46.466993 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.466904 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-host-var-lib-kubelet\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:38:46.466993 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.466946 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-etc-modprobe-d\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:38:46.466993 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.466960 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7ced550a-b3cb-4605-b8dc-aab3b55804b4-serviceca\") pod \"node-ca-p6hwx\" (UID: \"7ced550a-b3cb-4605-b8dc-aab3b55804b4\") " pod="openshift-image-registry/node-ca-p6hwx" Nov 03 20:38:46.467185 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.466996 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bafd96d7-6982-4c93-81fc-d61643002fd1-etc-selinux\") pod \"aws-ebs-csi-driver-node-jq7n8\" (UID: \"bafd96d7-6982-4c93-81fc-d61643002fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq7n8" Nov 03 20:38:46.467185 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.467027 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rlc8\" (UniqueName: \"kubernetes.io/projected/2dfb2512-b394-4a37-a5ba-a783d4e46c46-kube-api-access-5rlc8\") pod \"network-check-target-hc89p\" (UID: \"2dfb2512-b394-4a37-a5ba-a783d4e46c46\") " pod="openshift-network-diagnostics/network-check-target-hc89p" Nov 03 20:38:46.467185 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.467053 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-host-slash\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:38:46.467185 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.467027 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-multus-cni-dir\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:38:46.467185 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.467079 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-host-cni-netd\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:38:46.467185 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.467081 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bafd96d7-6982-4c93-81fc-d61643002fd1-etc-selinux\") pod \"aws-ebs-csi-driver-node-jq7n8\" (UID: \"bafd96d7-6982-4c93-81fc-d61643002fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq7n8" Nov 03 20:38:46.467185 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.467102 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/03985fe9-0dfe-4f9a-97db-bb4c0298be5c-kubelet-config\") pod \"global-pull-secret-syncer-g4892\" (UID: \"03985fe9-0dfe-4f9a-97db-bb4c0298be5c\") " pod="kube-system/global-pull-secret-syncer-g4892" Nov 03 20:38:46.467185 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.467129 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-host-run-k8s-cni-cncf-io\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:38:46.467185 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.467147 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/03985fe9-0dfe-4f9a-97db-bb4c0298be5c-kubelet-config\") pod \"global-pull-secret-syncer-g4892\" (UID: \"03985fe9-0dfe-4f9a-97db-bb4c0298be5c\") " pod="kube-system/global-pull-secret-syncer-g4892" Nov 03 20:38:46.467185 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.467148 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-host-slash\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:38:46.467185 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.467153 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bafd96d7-6982-4c93-81fc-d61643002fd1-device-dir\") pod \"aws-ebs-csi-driver-node-jq7n8\" (UID: \"bafd96d7-6982-4c93-81fc-d61643002fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq7n8" Nov 03 20:38:46.467185 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.467188 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f1f5ba6-ccc5-4f0d-a135-743747ce667a-cnibin\") pod \"multus-additional-cni-plugins-shzwn\" (UID: \"2f1f5ba6-ccc5-4f0d-a135-743747ce667a\") " pod="openshift-multus/multus-additional-cni-plugins-shzwn" Nov 03 20:38:46.467740 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.467148 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-host-cni-netd\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:38:46.467740 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.467219 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-host\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:38:46.467740 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.467187 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-host-run-k8s-cni-cncf-io\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:38:46.467740 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.467248 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-etc-tuned\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:38:46.467740 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.467253 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f1f5ba6-ccc5-4f0d-a135-743747ce667a-cnibin\") pod \"multus-additional-cni-plugins-shzwn\" (UID: \"2f1f5ba6-ccc5-4f0d-a135-743747ce667a\") " pod="openshift-multus/multus-additional-cni-plugins-shzwn" Nov 03 20:38:46.467740 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.467276 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-host-var-lib-cni-bin\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:38:46.467740 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.467285 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bafd96d7-6982-4c93-81fc-d61643002fd1-device-dir\") pod \"aws-ebs-csi-driver-node-jq7n8\" (UID: \"bafd96d7-6982-4c93-81fc-d61643002fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq7n8" Nov 03 20:38:46.467740 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.467303 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-run-systemd\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:38:46.467740 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.467313 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-host\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:38:46.467740 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.467327 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-log-socket\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:38:46.467740 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.467353 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/12658495-b5af-4c01-9c72-19b1173f7529-ovnkube-script-lib\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:38:46.467740 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.467356 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-host-var-lib-cni-bin\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:38:46.467740 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.467371 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-run-systemd\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:38:46.467740 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.467394 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-log-socket\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:38:46.467740 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.467416 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6e89ebac-2191-4acc-b4d4-ff326b3b74ac-iptables-alerter-script\") pod \"iptables-alerter-lvcbj\" (UID: \"6e89ebac-2191-4acc-b4d4-ff326b3b74ac\") " pod="openshift-network-operator/iptables-alerter-lvcbj" Nov 03 20:38:46.467740 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.467457 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2f1f5ba6-ccc5-4f0d-a135-743747ce667a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-shzwn\" (UID: \"2f1f5ba6-ccc5-4f0d-a135-743747ce667a\") " pod="openshift-multus/multus-additional-cni-plugins-shzwn" Nov 03 20:38:46.467740 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.467466 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-multus-socket-dir-parent\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:38:46.468549 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.467493 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25sxl\" (UniqueName: \"kubernetes.io/projected/bafd96d7-6982-4c93-81fc-d61643002fd1-kube-api-access-25sxl\") pod \"aws-ebs-csi-driver-node-jq7n8\" (UID: \"bafd96d7-6982-4c93-81fc-d61643002fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq7n8" Nov 03 20:38:46.468549 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.467515 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-systemd-units\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:38:46.468549 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.467517 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-multus-socket-dir-parent\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:38:46.468549 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.467547 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/12658495-b5af-4c01-9c72-19b1173f7529-ovn-node-metrics-cert\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:38:46.468549 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.467578 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z294f\" (UniqueName: \"kubernetes.io/projected/2f1f5ba6-ccc5-4f0d-a135-743747ce667a-kube-api-access-z294f\") pod \"multus-additional-cni-plugins-shzwn\" (UID: \"2f1f5ba6-ccc5-4f0d-a135-743747ce667a\") " pod="openshift-multus/multus-additional-cni-plugins-shzwn" Nov 03 20:38:46.468549 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.467557 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7ced550a-b3cb-4605-b8dc-aab3b55804b4-serviceca\") pod \"node-ca-p6hwx\" (UID: \"7ced550a-b3cb-4605-b8dc-aab3b55804b4\") " pod="openshift-image-registry/node-ca-p6hwx" Nov 03 20:38:46.468549 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.467587 2579 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 20:38:46.468549 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.467604 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-host-run-multus-certs\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:38:46.468549 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.467670 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-host-run-multus-certs\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:38:46.468549 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.467702 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-systemd-units\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:38:46.468549 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.467772 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bafd96d7-6982-4c93-81fc-d61643002fd1-registration-dir\") pod \"aws-ebs-csi-driver-node-jq7n8\" (UID: \"bafd96d7-6982-4c93-81fc-d61643002fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq7n8" Nov 03 20:38:46.468549 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.467803 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-host-cni-bin\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:38:46.468549 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.467829 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-etc-sysconfig\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:38:46.468549 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.467852 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-lib-modules\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:38:46.468549 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.467867 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bafd96d7-6982-4c93-81fc-d61643002fd1-registration-dir\") pod \"aws-ebs-csi-driver-node-jq7n8\" (UID: \"bafd96d7-6982-4c93-81fc-d61643002fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq7n8" Nov 03 20:38:46.468549 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.467877 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-hostroot\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:38:46.468549 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.467888 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-host-cni-bin\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:38:46.468549 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.467919 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-etc-sysconfig\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:38:46.469343 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.467951 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/12658495-b5af-4c01-9c72-19b1173f7529-ovnkube-script-lib\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:38:46.469343 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.467986 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-lib-modules\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:38:46.469343 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.468029 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4a5df9f4-aaaa-4441-a2e0-28744e197828-multus-daemon-config\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:38:46.469343 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.468033 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-hostroot\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:38:46.469343 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.468070 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2vcp\" (UniqueName: \"kubernetes.io/projected/4a5df9f4-aaaa-4441-a2e0-28744e197828-kube-api-access-z2vcp\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:38:46.469343 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.468087 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-host-kubelet\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:38:46.469343 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.468105 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:38:46.469343 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.468068 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6e89ebac-2191-4acc-b4d4-ff326b3b74ac-iptables-alerter-script\") pod \"iptables-alerter-lvcbj\" (UID: \"6e89ebac-2191-4acc-b4d4-ff326b3b74ac\") " pod="openshift-network-operator/iptables-alerter-lvcbj" Nov 03 20:38:46.469343 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.468128 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cf15bf7c-07c6-4d82-987b-aa4a332d5ca9-hosts-file\") pod \"node-resolver-gkm7b\" (UID: \"cf15bf7c-07c6-4d82-987b-aa4a332d5ca9\") " pod="openshift-dns/node-resolver-gkm7b" Nov 03 20:38:46.469343 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.468145 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8b6m\" (UniqueName: \"kubernetes.io/projected/cf15bf7c-07c6-4d82-987b-aa4a332d5ca9-kube-api-access-t8b6m\") pod \"node-resolver-gkm7b\" (UID: \"cf15bf7c-07c6-4d82-987b-aa4a332d5ca9\") " pod="openshift-dns/node-resolver-gkm7b" Nov 03 20:38:46.469343 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.468163 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-etc-sysctl-d\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:38:46.469343 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.468178 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-etc-sysctl-conf\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:38:46.469343 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.468202 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-tmp\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:38:46.469343 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.468220 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7ced550a-b3cb-4605-b8dc-aab3b55804b4-host\") pod \"node-ca-p6hwx\" (UID: \"7ced550a-b3cb-4605-b8dc-aab3b55804b4\") " pod="openshift-image-registry/node-ca-p6hwx" Nov 03 20:38:46.469343 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.468237 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-var-lib-kubelet\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:38:46.469343 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.468253 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3e72f317-1042-40b0-b709-dc3d650754b5-agent-certs\") pod \"konnectivity-agent-jvtt8\" (UID: \"3e72f317-1042-40b0-b709-dc3d650754b5\") " pod="kube-system/konnectivity-agent-jvtt8" Nov 03 20:38:46.469343 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.468259 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-host-kubelet\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:38:46.469343 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.468289 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3e72f317-1042-40b0-b709-dc3d650754b5-konnectivity-ca\") pod \"konnectivity-agent-jvtt8\" (UID: \"3e72f317-1042-40b0-b709-dc3d650754b5\") " pod="kube-system/konnectivity-agent-jvtt8" Nov 03 20:38:46.470181 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.468311 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f1f5ba6-ccc5-4f0d-a135-743747ce667a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-shzwn\" (UID: \"2f1f5ba6-ccc5-4f0d-a135-743747ce667a\") " pod="openshift-multus/multus-additional-cni-plugins-shzwn" Nov 03 20:38:46.470181 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.468328 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6e89ebac-2191-4acc-b4d4-ff326b3b74ac-host-slash\") pod \"iptables-alerter-lvcbj\" (UID: \"6e89ebac-2191-4acc-b4d4-ff326b3b74ac\") " pod="openshift-network-operator/iptables-alerter-lvcbj" Nov 03 20:38:46.470181 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.468325 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:38:46.470181 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.468341 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cf15bf7c-07c6-4d82-987b-aa4a332d5ca9-hosts-file\") pod \"node-resolver-gkm7b\" (UID: \"cf15bf7c-07c6-4d82-987b-aa4a332d5ca9\") " pod="openshift-dns/node-resolver-gkm7b" Nov 03 20:38:46.470181 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.468346 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-var-lib-openvswitch\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:38:46.470181 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.468469 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-etc-sysctl-d\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:38:46.470181 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.468538 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-var-lib-openvswitch\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:38:46.470181 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.468562 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6e89ebac-2191-4acc-b4d4-ff326b3b74ac-host-slash\") pod \"iptables-alerter-lvcbj\" (UID: \"6e89ebac-2191-4acc-b4d4-ff326b3b74ac\") " pod="openshift-network-operator/iptables-alerter-lvcbj" Nov 03 20:38:46.470181 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.468589 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4a5df9f4-aaaa-4441-a2e0-28744e197828-multus-daemon-config\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:38:46.470181 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.468647 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-var-lib-kubelet\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:38:46.470181 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.468653 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-etc-sysctl-conf\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:38:46.470181 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.468675 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7ced550a-b3cb-4605-b8dc-aab3b55804b4-host\") pod \"node-ca-p6hwx\" (UID: \"7ced550a-b3cb-4605-b8dc-aab3b55804b4\") " pod="openshift-image-registry/node-ca-p6hwx" Nov 03 20:38:46.470181 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.468690 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-etc-openvswitch\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:38:46.470181 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.468708 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-run-ovn\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:38:46.470181 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.468757 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmbq4\" (UniqueName: \"kubernetes.io/projected/6e89ebac-2191-4acc-b4d4-ff326b3b74ac-kube-api-access-tmbq4\") pod \"iptables-alerter-lvcbj\" (UID: \"6e89ebac-2191-4acc-b4d4-ff326b3b74ac\") " pod="openshift-network-operator/iptables-alerter-lvcbj" Nov 03 20:38:46.470181 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.468772 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-host-run-netns\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:38:46.470181 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.468804 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-multus-conf-dir\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:38:46.471101 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.468852 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-run-openvswitch\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:38:46.471101 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.468861 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f1f5ba6-ccc5-4f0d-a135-743747ce667a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-shzwn\" (UID: \"2f1f5ba6-ccc5-4f0d-a135-743747ce667a\") " pod="openshift-multus/multus-additional-cni-plugins-shzwn" Nov 03 20:38:46.471101 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.468905 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf15bf7c-07c6-4d82-987b-aa4a332d5ca9-tmp-dir\") pod \"node-resolver-gkm7b\" (UID: \"cf15bf7c-07c6-4d82-987b-aa4a332d5ca9\") " pod="openshift-dns/node-resolver-gkm7b" Nov 03 20:38:46.471101 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.468961 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/03985fe9-0dfe-4f9a-97db-bb4c0298be5c-dbus\") pod \"global-pull-secret-syncer-g4892\" (UID: \"03985fe9-0dfe-4f9a-97db-bb4c0298be5c\") " pod="kube-system/global-pull-secret-syncer-g4892" Nov 03 20:38:46.471101 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.468989 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-host-run-netns\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:38:46.471101 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.469056 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-run-ovn\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:38:46.471101 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.469061 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-run-openvswitch\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:38:46.471101 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.469083 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3e72f317-1042-40b0-b709-dc3d650754b5-konnectivity-ca\") pod \"konnectivity-agent-jvtt8\" (UID: \"3e72f317-1042-40b0-b709-dc3d650754b5\") " pod="kube-system/konnectivity-agent-jvtt8" Nov 03 20:38:46.471101 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.469125 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-cnibin\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:38:46.471101 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.469202 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-multus-conf-dir\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:38:46.471101 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.469261 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/03985fe9-0dfe-4f9a-97db-bb4c0298be5c-dbus\") pod \"global-pull-secret-syncer-g4892\" (UID: \"03985fe9-0dfe-4f9a-97db-bb4c0298be5c\") " pod="kube-system/global-pull-secret-syncer-g4892" Nov 03 20:38:46.471101 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.469265 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-host-var-lib-cni-multus\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:38:46.471101 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.469291 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-host-var-lib-cni-multus\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:38:46.471101 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.469336 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bafd96d7-6982-4c93-81fc-d61643002fd1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jq7n8\" (UID: \"bafd96d7-6982-4c93-81fc-d61643002fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq7n8" Nov 03 20:38:46.471101 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.469378 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bafd96d7-6982-4c93-81fc-d61643002fd1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jq7n8\" (UID: \"bafd96d7-6982-4c93-81fc-d61643002fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq7n8" Nov 03 20:38:46.471101 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.469462 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/12658495-b5af-4c01-9c72-19b1173f7529-env-overrides\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:38:46.471101 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.469473 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf15bf7c-07c6-4d82-987b-aa4a332d5ca9-tmp-dir\") pod \"node-resolver-gkm7b\" (UID: \"cf15bf7c-07c6-4d82-987b-aa4a332d5ca9\") " pod="openshift-dns/node-resolver-gkm7b" Nov 03 20:38:46.471101 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.469519 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-sys\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:38:46.471769 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.469552 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-cnibin\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:38:46.471769 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.469593 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj57q\" (UniqueName: \"kubernetes.io/projected/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-kube-api-access-wj57q\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:38:46.471769 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.469631 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bafd96d7-6982-4c93-81fc-d61643002fd1-sys-fs\") pod \"aws-ebs-csi-driver-node-jq7n8\" (UID: \"bafd96d7-6982-4c93-81fc-d61643002fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq7n8" Nov 03 20:38:46.471769 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.469652 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-node-log\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:38:46.471769 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.469684 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/12658495-b5af-4c01-9c72-19b1173f7529-ovnkube-config\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:38:46.471769 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.469711 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03985fe9-0dfe-4f9a-97db-bb4c0298be5c-original-pull-secret\") pod \"global-pull-secret-syncer-g4892\" (UID: \"03985fe9-0dfe-4f9a-97db-bb4c0298be5c\") " pod="kube-system/global-pull-secret-syncer-g4892" Nov 03 20:38:46.471769 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.469720 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-sys\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:38:46.471769 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.469741 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-etc-kubernetes\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:38:46.471769 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.469774 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bafd96d7-6982-4c93-81fc-d61643002fd1-sys-fs\") pod \"aws-ebs-csi-driver-node-jq7n8\" (UID: \"bafd96d7-6982-4c93-81fc-d61643002fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq7n8" Nov 03 20:38:46.471769 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.469857 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/12658495-b5af-4c01-9c72-19b1173f7529-env-overrides\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:38:46.471769 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:46.469877 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 20:38:46.471769 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.469909 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-run\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:38:46.471769 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.469944 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-node-log\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:38:46.471769 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:46.469952 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03985fe9-0dfe-4f9a-97db-bb4c0298be5c-original-pull-secret podName:03985fe9-0dfe-4f9a-97db-bb4c0298be5c nodeName:}" failed. No retries permitted until 2026-11-03 20:38:46.969929669 +0000 UTC m=+2.274264719 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/03985fe9-0dfe-4f9a-97db-bb4c0298be5c-original-pull-secret") pod "global-pull-secret-syncer-g4892" (UID: "03985fe9-0dfe-4f9a-97db-bb4c0298be5c") : object "kube-system"/"original-pull-secret" not registered Nov 03 20:38:46.471769 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.469969 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-system-cni-dir\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:38:46.471769 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.469996 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bafd96d7-6982-4c93-81fc-d61643002fd1-socket-dir\") pod \"aws-ebs-csi-driver-node-jq7n8\" (UID: \"bafd96d7-6982-4c93-81fc-d61643002fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq7n8" Nov 03 20:38:46.471769 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.470011 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-run\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:38:46.472446 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.470046 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-system-cni-dir\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:38:46.472446 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.470048 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-etc-kubernetes\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:38:46.472446 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.470133 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bafd96d7-6982-4c93-81fc-d61643002fd1-socket-dir\") pod \"aws-ebs-csi-driver-node-jq7n8\" (UID: \"bafd96d7-6982-4c93-81fc-d61643002fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq7n8" Nov 03 20:38:46.472446 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.470190 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-host-run-netns\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:38:46.472446 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.470234 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cscp\" (UniqueName: \"kubernetes.io/projected/12658495-b5af-4c01-9c72-19b1173f7529-kube-api-access-6cscp\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:38:46.472446 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.470266 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-host-run-netns\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:38:46.472446 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.470295 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-etc-openvswitch\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:38:46.472446 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.470321 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/12658495-b5af-4c01-9c72-19b1173f7529-ovnkube-config\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:38:46.472446 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.470324 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2f1f5ba6-ccc5-4f0d-a135-743747ce667a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-shzwn\" (UID: \"2f1f5ba6-ccc5-4f0d-a135-743747ce667a\") " pod="openshift-multus/multus-additional-cni-plugins-shzwn" Nov 03 20:38:46.472446 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.470485 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-etc-systemd\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:38:46.472446 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.470519 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f1f5ba6-ccc5-4f0d-a135-743747ce667a-system-cni-dir\") pod \"multus-additional-cni-plugins-shzwn\" (UID: \"2f1f5ba6-ccc5-4f0d-a135-743747ce667a\") " pod="openshift-multus/multus-additional-cni-plugins-shzwn" Nov 03 20:38:46.472446 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.470565 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f1f5ba6-ccc5-4f0d-a135-743747ce667a-cni-binary-copy\") pod \"multus-additional-cni-plugins-shzwn\" (UID: \"2f1f5ba6-ccc5-4f0d-a135-743747ce667a\") " pod="openshift-multus/multus-additional-cni-plugins-shzwn" Nov 03 20:38:46.472446 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.470618 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4a5df9f4-aaaa-4441-a2e0-28744e197828-cni-binary-copy\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:38:46.472446 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.470662 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6rjq\" (UniqueName: \"kubernetes.io/projected/7ced550a-b3cb-4605-b8dc-aab3b55804b4-kube-api-access-t6rjq\") pod \"node-ca-p6hwx\" (UID: \"7ced550a-b3cb-4605-b8dc-aab3b55804b4\") " pod="openshift-image-registry/node-ca-p6hwx" Nov 03 20:38:46.472446 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.470662 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f1f5ba6-ccc5-4f0d-a135-743747ce667a-system-cni-dir\") pod \"multus-additional-cni-plugins-shzwn\" (UID: \"2f1f5ba6-ccc5-4f0d-a135-743747ce667a\") " pod="openshift-multus/multus-additional-cni-plugins-shzwn" Nov 03 20:38:46.472446 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.470717 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-etc-systemd\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:38:46.472446 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.470903 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/747345eb-a407-423b-92df-c5d765a7da2d-metrics-certs\") pod \"network-metrics-daemon-xt5qx\" (UID: \"747345eb-a407-423b-92df-c5d765a7da2d\") " pod="openshift-multus/network-metrics-daemon-xt5qx" Nov 03 20:38:46.473266 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.470944 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2brl\" (UniqueName: \"kubernetes.io/projected/747345eb-a407-423b-92df-c5d765a7da2d-kube-api-access-l2brl\") pod \"network-metrics-daemon-xt5qx\" (UID: \"747345eb-a407-423b-92df-c5d765a7da2d\") " pod="openshift-multus/network-metrics-daemon-xt5qx" Nov 03 20:38:46.473266 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.470971 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-host-run-ovn-kubernetes\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:38:46.473266 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.471046 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-host-run-ovn-kubernetes\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:38:46.473266 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:46.472088 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:38:46.473266 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:46.472175 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/747345eb-a407-423b-92df-c5d765a7da2d-metrics-certs podName:747345eb-a407-423b-92df-c5d765a7da2d nodeName:}" failed. No retries permitted until 2026-11-03 20:38:46.972155817 +0000 UTC m=+2.276490883 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/747345eb-a407-423b-92df-c5d765a7da2d-metrics-certs") pod "network-metrics-daemon-xt5qx" (UID: "747345eb-a407-423b-92df-c5d765a7da2d") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:38:46.473266 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.472838 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4a5df9f4-aaaa-4441-a2e0-28744e197828-cni-binary-copy\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:38:46.473266 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.473072 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3e72f317-1042-40b0-b709-dc3d650754b5-agent-certs\") pod \"konnectivity-agent-jvtt8\" (UID: \"3e72f317-1042-40b0-b709-dc3d650754b5\") " pod="kube-system/konnectivity-agent-jvtt8" Nov 03 20:38:46.473266 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.473195 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-tmp\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:38:46.473266 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.473205 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/12658495-b5af-4c01-9c72-19b1173f7529-ovn-node-metrics-cert\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:38:46.473266 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.473243 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-etc-tuned\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:38:46.474033 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.474009 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f1f5ba6-ccc5-4f0d-a135-743747ce667a-cni-binary-copy\") pod \"multus-additional-cni-plugins-shzwn\" (UID: \"2f1f5ba6-ccc5-4f0d-a135-743747ce667a\") " pod="openshift-multus/multus-additional-cni-plugins-shzwn" Nov 03 20:38:46.474166 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.474032 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2f1f5ba6-ccc5-4f0d-a135-743747ce667a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-shzwn\" (UID: \"2f1f5ba6-ccc5-4f0d-a135-743747ce667a\") " pod="openshift-multus/multus-additional-cni-plugins-shzwn" Nov 03 20:38:46.527760 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:46.527697 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 20:38:46.527760 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:46.527724 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 20:38:46.527760 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:46.527736 2579 projected.go:196] Error preparing data for projected volume kube-api-access-5rlc8 for pod openshift-network-diagnostics/network-check-target-hc89p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:38:46.527970 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:46.527812 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2dfb2512-b394-4a37-a5ba-a783d4e46c46-kube-api-access-5rlc8 podName:2dfb2512-b394-4a37-a5ba-a783d4e46c46 nodeName:}" failed. No retries permitted until 2026-11-03 20:38:47.027795366 +0000 UTC m=+2.332130431 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-5rlc8" (UniqueName: "kubernetes.io/projected/2dfb2512-b394-4a37-a5ba-a783d4e46c46-kube-api-access-5rlc8") pod "network-check-target-hc89p" (UID: "2dfb2512-b394-4a37-a5ba-a783d4e46c46") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:38:46.530057 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.530025 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z294f\" (UniqueName: \"kubernetes.io/projected/2f1f5ba6-ccc5-4f0d-a135-743747ce667a-kube-api-access-z294f\") pod \"multus-additional-cni-plugins-shzwn\" (UID: \"2f1f5ba6-ccc5-4f0d-a135-743747ce667a\") " pod="openshift-multus/multus-additional-cni-plugins-shzwn" Nov 03 20:38:46.530481 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.530460 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2vcp\" (UniqueName: \"kubernetes.io/projected/4a5df9f4-aaaa-4441-a2e0-28744e197828-kube-api-access-z2vcp\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:38:46.530585 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.530515 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25sxl\" (UniqueName: \"kubernetes.io/projected/bafd96d7-6982-4c93-81fc-d61643002fd1-kube-api-access-25sxl\") pod \"aws-ebs-csi-driver-node-jq7n8\" (UID: \"bafd96d7-6982-4c93-81fc-d61643002fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq7n8" Nov 03 20:38:46.531259 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.531238 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmbq4\" (UniqueName: \"kubernetes.io/projected/6e89ebac-2191-4acc-b4d4-ff326b3b74ac-kube-api-access-tmbq4\") pod \"iptables-alerter-lvcbj\" (UID: \"6e89ebac-2191-4acc-b4d4-ff326b3b74ac\") " pod="openshift-network-operator/iptables-alerter-lvcbj" Nov 03 20:38:46.533369 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.533340 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj57q\" (UniqueName: \"kubernetes.io/projected/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-kube-api-access-wj57q\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:38:46.533369 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.533360 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cscp\" (UniqueName: \"kubernetes.io/projected/12658495-b5af-4c01-9c72-19b1173f7529-kube-api-access-6cscp\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:38:46.534229 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.534208 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2brl\" (UniqueName: \"kubernetes.io/projected/747345eb-a407-423b-92df-c5d765a7da2d-kube-api-access-l2brl\") pod \"network-metrics-daemon-xt5qx\" (UID: \"747345eb-a407-423b-92df-c5d765a7da2d\") " pod="openshift-multus/network-metrics-daemon-xt5qx" Nov 03 20:38:46.535831 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.535811 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6rjq\" (UniqueName: \"kubernetes.io/projected/7ced550a-b3cb-4605-b8dc-aab3b55804b4-kube-api-access-t6rjq\") pod \"node-ca-p6hwx\" (UID: \"7ced550a-b3cb-4605-b8dc-aab3b55804b4\") " pod="openshift-image-registry/node-ca-p6hwx" Nov 03 20:38:46.536832 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.536798 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8b6m\" (UniqueName: \"kubernetes.io/projected/cf15bf7c-07c6-4d82-987b-aa4a332d5ca9-kube-api-access-t8b6m\") pod \"node-resolver-gkm7b\" (UID: \"cf15bf7c-07c6-4d82-987b-aa4a332d5ca9\") " pod="openshift-dns/node-resolver-gkm7b" Nov 03 20:38:46.637927 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.637894 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:38:46.656356 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.656324 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jvtt8" Nov 03 20:38:46.671258 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.671227 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4k6r2" Nov 03 20:38:46.692496 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.692461 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:38:46.692620 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.692509 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gkm7b" Nov 03 20:38:46.695115 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.695093 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-p6hwx" Nov 03 20:38:46.717413 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:46.717375 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a5df9f4_aaaa_4441_a2e0_28744e197828.slice/crio-0244c122743d370f3b669ba61dda5e3b66c9cbafc104f327f0f7319575e86e30 WatchSource:0}: Error finding container 0244c122743d370f3b669ba61dda5e3b66c9cbafc104f327f0f7319575e86e30: Status 404 returned error can't find the container with id 0244c122743d370f3b669ba61dda5e3b66c9cbafc104f327f0f7319575e86e30 Nov 03 20:38:46.718548 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:46.718521 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6e5ca9eba1ab368215b5511cd840a25.slice/crio-90c4be718888538813c8970ee006a8aacbc94a7b254ed0aa3bc6b4bdcb72b2be WatchSource:0}: Error finding container 90c4be718888538813c8970ee006a8aacbc94a7b254ed0aa3bc6b4bdcb72b2be: Status 404 returned error can't find the container with id 90c4be718888538813c8970ee006a8aacbc94a7b254ed0aa3bc6b4bdcb72b2be Nov 03 20:38:46.721371 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:46.721346 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12658495_b5af_4c01_9c72_19b1173f7529.slice/crio-ad5cd83d0b08d50512686eab4cbec286050d49546867978894e38f67ed35fc74 WatchSource:0}: Error finding container ad5cd83d0b08d50512686eab4cbec286050d49546867978894e38f67ed35fc74: Status 404 returned error can't find the container with id ad5cd83d0b08d50512686eab4cbec286050d49546867978894e38f67ed35fc74 Nov 03 20:38:46.724584 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.724565 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:38:46.729731 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:46.729706 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf15bf7c_07c6_4d82_987b_aa4a332d5ca9.slice/crio-0fb1e6fcef76d2b173be8c0c5306a824f9f80d463e0267b5c4299accdb3b189e WatchSource:0}: Error finding container 0fb1e6fcef76d2b173be8c0c5306a824f9f80d463e0267b5c4299accdb3b189e: Status 404 returned error can't find the container with id 0fb1e6fcef76d2b173be8c0c5306a824f9f80d463e0267b5c4299accdb3b189e Nov 03 20:38:46.730451 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:46.730425 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6873aa06186bc61b65aa0fa7f8c6d9a2.slice/crio-fc504e8ae03bf81d454277ab0a6d074765ef9f32dcb03c2bb8d8b38f2e759ba3 WatchSource:0}: Error finding container fc504e8ae03bf81d454277ab0a6d074765ef9f32dcb03c2bb8d8b38f2e759ba3: Status 404 returned error can't find the container with id fc504e8ae03bf81d454277ab0a6d074765ef9f32dcb03c2bb8d8b38f2e759ba3 Nov 03 20:38:46.734680 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.734661 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lvcbj" Nov 03 20:38:46.740027 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.740001 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-shzwn" Nov 03 20:38:46.742236 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.742213 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq7n8" Nov 03 20:38:46.745396 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:46.745377 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e89ebac_2191_4acc_b4d4_ff326b3b74ac.slice/crio-6295278154ec9f50f8125c2ddc58308bc6b1e57f7a7db0f1e6a0d3cc5a26b298 WatchSource:0}: Error finding container 6295278154ec9f50f8125c2ddc58308bc6b1e57f7a7db0f1e6a0d3cc5a26b298: Status 404 returned error can't find the container with id 6295278154ec9f50f8125c2ddc58308bc6b1e57f7a7db0f1e6a0d3cc5a26b298 Nov 03 20:38:46.751696 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:46.751673 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f1f5ba6_ccc5_4f0d_a135_743747ce667a.slice/crio-335f16e7434a1f6f00cadb6dafbd0ed781f13d71b72ded6c24a31004d704cbf3 WatchSource:0}: Error finding container 335f16e7434a1f6f00cadb6dafbd0ed781f13d71b72ded6c24a31004d704cbf3: Status 404 returned error can't find the container with id 335f16e7434a1f6f00cadb6dafbd0ed781f13d71b72ded6c24a31004d704cbf3 Nov 03 20:38:46.752341 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:46.752317 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbafd96d7_6982_4c93_81fc_d61643002fd1.slice/crio-f8d26f4dc3ef29cbb33bb1657db2beddcc5187dd78248b6593e7c3f469d4499e WatchSource:0}: Error finding container f8d26f4dc3ef29cbb33bb1657db2beddcc5187dd78248b6593e7c3f469d4499e: Status 404 returned error can't find the container with id f8d26f4dc3ef29cbb33bb1657db2beddcc5187dd78248b6593e7c3f469d4499e Nov 03 20:38:46.975303 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.975215 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03985fe9-0dfe-4f9a-97db-bb4c0298be5c-original-pull-secret\") pod \"global-pull-secret-syncer-g4892\" (UID: \"03985fe9-0dfe-4f9a-97db-bb4c0298be5c\") " pod="kube-system/global-pull-secret-syncer-g4892" Nov 03 20:38:46.975303 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:46.975252 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/747345eb-a407-423b-92df-c5d765a7da2d-metrics-certs\") pod \"network-metrics-daemon-xt5qx\" (UID: \"747345eb-a407-423b-92df-c5d765a7da2d\") " pod="openshift-multus/network-metrics-daemon-xt5qx" Nov 03 20:38:46.975559 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:46.975366 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:38:46.975559 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:46.975409 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 20:38:46.975559 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:46.975419 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/747345eb-a407-423b-92df-c5d765a7da2d-metrics-certs podName:747345eb-a407-423b-92df-c5d765a7da2d nodeName:}" failed. No retries permitted until 2026-11-03 20:38:47.975404722 +0000 UTC m=+3.279739771 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/747345eb-a407-423b-92df-c5d765a7da2d-metrics-certs") pod "network-metrics-daemon-xt5qx" (UID: "747345eb-a407-423b-92df-c5d765a7da2d") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:38:46.975559 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:46.975512 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03985fe9-0dfe-4f9a-97db-bb4c0298be5c-original-pull-secret podName:03985fe9-0dfe-4f9a-97db-bb4c0298be5c nodeName:}" failed. No retries permitted until 2026-11-03 20:38:47.975490748 +0000 UTC m=+3.279825799 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/03985fe9-0dfe-4f9a-97db-bb4c0298be5c-original-pull-secret") pod "global-pull-secret-syncer-g4892" (UID: "03985fe9-0dfe-4f9a-97db-bb4c0298be5c") : object "kube-system"/"original-pull-secret" not registered Nov 03 20:38:47.076421 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:47.076379 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rlc8\" (UniqueName: \"kubernetes.io/projected/2dfb2512-b394-4a37-a5ba-a783d4e46c46-kube-api-access-5rlc8\") pod \"network-check-target-hc89p\" (UID: \"2dfb2512-b394-4a37-a5ba-a783d4e46c46\") " pod="openshift-network-diagnostics/network-check-target-hc89p" Nov 03 20:38:47.077129 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:47.076555 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 20:38:47.077129 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:47.076575 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 20:38:47.077129 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:47.076586 2579 projected.go:196] Error preparing data for projected volume kube-api-access-5rlc8 for pod openshift-network-diagnostics/network-check-target-hc89p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:38:47.077129 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:47.076651 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2dfb2512-b394-4a37-a5ba-a783d4e46c46-kube-api-access-5rlc8 podName:2dfb2512-b394-4a37-a5ba-a783d4e46c46 nodeName:}" failed. No retries permitted until 2026-11-03 20:38:48.076628168 +0000 UTC m=+3.380963234 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-5rlc8" (UniqueName: "kubernetes.io/projected/2dfb2512-b394-4a37-a5ba-a783d4e46c46-kube-api-access-5rlc8") pod "network-check-target-hc89p" (UID: "2dfb2512-b394-4a37-a5ba-a783d4e46c46") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:38:47.185346 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:47.185280 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gkm7b" event={"ID":"cf15bf7c-07c6-4d82-987b-aa4a332d5ca9","Type":"ContainerStarted","Data":"0fb1e6fcef76d2b173be8c0c5306a824f9f80d463e0267b5c4299accdb3b189e"} Nov 03 20:38:47.186388 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:47.186362 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-p6hwx" event={"ID":"7ced550a-b3cb-4605-b8dc-aab3b55804b4","Type":"ContainerStarted","Data":"81847b395d92df6e6a5c82d000e7362957027e3524f065444da36d85198c98db"} Nov 03 20:38:47.187449 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:47.187417 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-175.us-west-1.compute.internal" event={"ID":"c6e5ca9eba1ab368215b5511cd840a25","Type":"ContainerStarted","Data":"90c4be718888538813c8970ee006a8aacbc94a7b254ed0aa3bc6b4bdcb72b2be"} Nov 03 20:38:47.188736 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:47.188715 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" event={"ID":"12658495-b5af-4c01-9c72-19b1173f7529","Type":"ContainerStarted","Data":"ad5cd83d0b08d50512686eab4cbec286050d49546867978894e38f67ed35fc74"} Nov 03 20:38:47.189890 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:47.189869 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4k6r2" event={"ID":"4a5df9f4-aaaa-4441-a2e0-28744e197828","Type":"ContainerStarted","Data":"0244c122743d370f3b669ba61dda5e3b66c9cbafc104f327f0f7319575e86e30"} Nov 03 20:38:47.190992 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:47.190959 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq7n8" event={"ID":"bafd96d7-6982-4c93-81fc-d61643002fd1","Type":"ContainerStarted","Data":"f8d26f4dc3ef29cbb33bb1657db2beddcc5187dd78248b6593e7c3f469d4499e"} Nov 03 20:38:47.192186 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:47.192165 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-shzwn" event={"ID":"2f1f5ba6-ccc5-4f0d-a135-743747ce667a","Type":"ContainerStarted","Data":"335f16e7434a1f6f00cadb6dafbd0ed781f13d71b72ded6c24a31004d704cbf3"} Nov 03 20:38:47.193360 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:47.193341 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lvcbj" event={"ID":"6e89ebac-2191-4acc-b4d4-ff326b3b74ac","Type":"ContainerStarted","Data":"6295278154ec9f50f8125c2ddc58308bc6b1e57f7a7db0f1e6a0d3cc5a26b298"} Nov 03 20:38:47.194782 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:47.194754 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-175.us-west-1.compute.internal" event={"ID":"6873aa06186bc61b65aa0fa7f8c6d9a2","Type":"ContainerStarted","Data":"fc504e8ae03bf81d454277ab0a6d074765ef9f32dcb03c2bb8d8b38f2e759ba3"} Nov 03 20:38:47.248550 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:47.248457 2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:38:47.356382 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:47.356349 2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:38:47.384756 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:47.384725 2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:38:47.461848 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:47.461770 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 20:33:47 +0000 UTC" deadline="2028-06-21 05:46:03.572421488 +0000 UTC" Nov 03 20:38:47.461848 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:47.461840 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14289h7m16.110584344s" Nov 03 20:38:47.515071 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:47.515042 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e72f317_1042_40b0_b709_dc3d650754b5.slice/crio-d7d2df7f30348457242061c096e76a42dad7a50762bf9b4945e6f2e08467c118 WatchSource:0}: Error finding container d7d2df7f30348457242061c096e76a42dad7a50762bf9b4945e6f2e08467c118: Status 404 returned error can't find the container with id d7d2df7f30348457242061c096e76a42dad7a50762bf9b4945e6f2e08467c118 Nov 03 20:38:47.522565 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:38:47.522537 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d81822a_bda4_4f5e_a5f4_05e1606f22bf.slice/crio-79db833470e5e69ef9f9af3aa5e17c2706b66132c8a0593597ae180159c9894d WatchSource:0}: Error finding container 79db833470e5e69ef9f9af3aa5e17c2706b66132c8a0593597ae180159c9894d: Status 404 returned error can't find the container with id 79db833470e5e69ef9f9af3aa5e17c2706b66132c8a0593597ae180159c9894d Nov 03 20:38:47.984846 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:47.984749 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03985fe9-0dfe-4f9a-97db-bb4c0298be5c-original-pull-secret\") pod \"global-pull-secret-syncer-g4892\" (UID: \"03985fe9-0dfe-4f9a-97db-bb4c0298be5c\") " pod="kube-system/global-pull-secret-syncer-g4892" Nov 03 20:38:47.984846 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:47.984807 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/747345eb-a407-423b-92df-c5d765a7da2d-metrics-certs\") pod \"network-metrics-daemon-xt5qx\" (UID: \"747345eb-a407-423b-92df-c5d765a7da2d\") " pod="openshift-multus/network-metrics-daemon-xt5qx" Nov 03 20:38:47.985168 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:47.984959 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:38:47.985168 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:47.985022 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/747345eb-a407-423b-92df-c5d765a7da2d-metrics-certs podName:747345eb-a407-423b-92df-c5d765a7da2d nodeName:}" failed. No retries permitted until 2026-11-03 20:38:49.985002836 +0000 UTC m=+5.289337890 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/747345eb-a407-423b-92df-c5d765a7da2d-metrics-certs") pod "network-metrics-daemon-xt5qx" (UID: "747345eb-a407-423b-92df-c5d765a7da2d") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:38:47.985168 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:47.985088 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 20:38:47.985168 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:47.985125 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03985fe9-0dfe-4f9a-97db-bb4c0298be5c-original-pull-secret podName:03985fe9-0dfe-4f9a-97db-bb4c0298be5c nodeName:}" failed. No retries permitted until 2026-11-03 20:38:49.985113544 +0000 UTC m=+5.289448597 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/03985fe9-0dfe-4f9a-97db-bb4c0298be5c-original-pull-secret") pod "global-pull-secret-syncer-g4892" (UID: "03985fe9-0dfe-4f9a-97db-bb4c0298be5c") : object "kube-system"/"original-pull-secret" not registered Nov 03 20:38:48.085601 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:48.085555 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rlc8\" (UniqueName: \"kubernetes.io/projected/2dfb2512-b394-4a37-a5ba-a783d4e46c46-kube-api-access-5rlc8\") pod \"network-check-target-hc89p\" (UID: \"2dfb2512-b394-4a37-a5ba-a783d4e46c46\") " pod="openshift-network-diagnostics/network-check-target-hc89p" Nov 03 20:38:48.086190 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:48.085742 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 20:38:48.086190 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:48.085767 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 20:38:48.086190 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:48.085781 2579 projected.go:196] Error preparing data for projected volume kube-api-access-5rlc8 for pod openshift-network-diagnostics/network-check-target-hc89p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:38:48.086190 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:48.085842 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2dfb2512-b394-4a37-a5ba-a783d4e46c46-kube-api-access-5rlc8 podName:2dfb2512-b394-4a37-a5ba-a783d4e46c46 nodeName:}" failed. No retries permitted until 2026-11-03 20:38:50.085823655 +0000 UTC m=+5.390158726 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-5rlc8" (UniqueName: "kubernetes.io/projected/2dfb2512-b394-4a37-a5ba-a783d4e46c46-kube-api-access-5rlc8") pod "network-check-target-hc89p" (UID: "2dfb2512-b394-4a37-a5ba-a783d4e46c46") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:38:48.180741 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:48.180703 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g4892" Nov 03 20:38:48.180933 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:48.180852 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g4892" podUID="03985fe9-0dfe-4f9a-97db-bb4c0298be5c" Nov 03 20:38:48.180933 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:48.180927 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xt5qx" Nov 03 20:38:48.181049 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:48.180988 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xt5qx" podUID="747345eb-a407-423b-92df-c5d765a7da2d" Nov 03 20:38:48.181049 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:48.181032 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hc89p" Nov 03 20:38:48.181153 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:48.181090 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hc89p" podUID="2dfb2512-b394-4a37-a5ba-a783d4e46c46" Nov 03 20:38:48.210742 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:48.210693 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jvtt8" event={"ID":"3e72f317-1042-40b0-b709-dc3d650754b5","Type":"ContainerStarted","Data":"d7d2df7f30348457242061c096e76a42dad7a50762bf9b4945e6f2e08467c118"} Nov 03 20:38:48.221450 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:48.221412 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" event={"ID":"6d81822a-bda4-4f5e-a5f4-05e1606f22bf","Type":"ContainerStarted","Data":"79db833470e5e69ef9f9af3aa5e17c2706b66132c8a0593597ae180159c9894d"} Nov 03 20:38:48.463073 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:48.462946 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 20:33:47 +0000 UTC" deadline="2028-05-03 01:47:20.747365596 +0000 UTC" Nov 03 20:38:48.463073 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:48.463025 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13109h8m32.284345209s" Nov 03 20:38:50.002563 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:50.002492 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03985fe9-0dfe-4f9a-97db-bb4c0298be5c-original-pull-secret\") pod \"global-pull-secret-syncer-g4892\" (UID: \"03985fe9-0dfe-4f9a-97db-bb4c0298be5c\") " pod="kube-system/global-pull-secret-syncer-g4892" Nov 03 20:38:50.003216 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:50.002800 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/747345eb-a407-423b-92df-c5d765a7da2d-metrics-certs\") pod \"network-metrics-daemon-xt5qx\" (UID: \"747345eb-a407-423b-92df-c5d765a7da2d\") " pod="openshift-multus/network-metrics-daemon-xt5qx" Nov 03 20:38:50.003216 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:50.002651 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 20:38:50.003216 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:50.002989 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03985fe9-0dfe-4f9a-97db-bb4c0298be5c-original-pull-secret podName:03985fe9-0dfe-4f9a-97db-bb4c0298be5c nodeName:}" failed. No retries permitted until 2026-11-03 20:38:54.002970154 +0000 UTC m=+9.307305218 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/03985fe9-0dfe-4f9a-97db-bb4c0298be5c-original-pull-secret") pod "global-pull-secret-syncer-g4892" (UID: "03985fe9-0dfe-4f9a-97db-bb4c0298be5c") : object "kube-system"/"original-pull-secret" not registered Nov 03 20:38:50.003216 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:50.002923 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:38:50.003216 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:50.003038 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/747345eb-a407-423b-92df-c5d765a7da2d-metrics-certs podName:747345eb-a407-423b-92df-c5d765a7da2d nodeName:}" failed. No retries permitted until 2026-11-03 20:38:54.003026662 +0000 UTC m=+9.307361721 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/747345eb-a407-423b-92df-c5d765a7da2d-metrics-certs") pod "network-metrics-daemon-xt5qx" (UID: "747345eb-a407-423b-92df-c5d765a7da2d") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:38:50.104212 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:50.103990 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rlc8\" (UniqueName: \"kubernetes.io/projected/2dfb2512-b394-4a37-a5ba-a783d4e46c46-kube-api-access-5rlc8\") pod \"network-check-target-hc89p\" (UID: \"2dfb2512-b394-4a37-a5ba-a783d4e46c46\") " pod="openshift-network-diagnostics/network-check-target-hc89p" Nov 03 20:38:50.104212 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:50.104200 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 20:38:50.104212 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:50.104224 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 20:38:50.104560 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:50.104237 2579 projected.go:196] Error preparing data for projected volume kube-api-access-5rlc8 for pod openshift-network-diagnostics/network-check-target-hc89p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:38:50.104560 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:50.104299 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2dfb2512-b394-4a37-a5ba-a783d4e46c46-kube-api-access-5rlc8 podName:2dfb2512-b394-4a37-a5ba-a783d4e46c46 nodeName:}" failed. No retries permitted until 2026-11-03 20:38:54.104279692 +0000 UTC m=+9.408614749 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-5rlc8" (UniqueName: "kubernetes.io/projected/2dfb2512-b394-4a37-a5ba-a783d4e46c46-kube-api-access-5rlc8") pod "network-check-target-hc89p" (UID: "2dfb2512-b394-4a37-a5ba-a783d4e46c46") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:38:50.181015 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:50.180807 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xt5qx" Nov 03 20:38:50.181015 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:50.180818 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hc89p" Nov 03 20:38:50.181015 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:50.180962 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xt5qx" podUID="747345eb-a407-423b-92df-c5d765a7da2d" Nov 03 20:38:50.181380 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:50.181030 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hc89p" podUID="2dfb2512-b394-4a37-a5ba-a783d4e46c46" Nov 03 20:38:50.181380 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:50.180821 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g4892" Nov 03 20:38:50.181380 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:50.181132 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g4892" podUID="03985fe9-0dfe-4f9a-97db-bb4c0298be5c" Nov 03 20:38:52.180631 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:52.180334 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hc89p" Nov 03 20:38:52.181070 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:52.180749 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hc89p" podUID="2dfb2512-b394-4a37-a5ba-a783d4e46c46" Nov 03 20:38:52.181070 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:52.180837 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g4892" Nov 03 20:38:52.181070 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:52.180914 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g4892" podUID="03985fe9-0dfe-4f9a-97db-bb4c0298be5c" Nov 03 20:38:52.181070 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:52.180984 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xt5qx" Nov 03 20:38:52.181070 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:52.181065 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xt5qx" podUID="747345eb-a407-423b-92df-c5d765a7da2d" Nov 03 20:38:52.256436 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:52.256397 2579 generic.go:356] "Generic (PLEG): container finished" podID="2f1f5ba6-ccc5-4f0d-a135-743747ce667a" containerID="3f3a9d69b49c65f870f7de2826852bf1e1c69765b95d32cc848a28571f3ad64e" exitCode=0 Nov 03 20:38:52.256649 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:52.256533 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-shzwn" event={"ID":"2f1f5ba6-ccc5-4f0d-a135-743747ce667a","Type":"ContainerDied","Data":"3f3a9d69b49c65f870f7de2826852bf1e1c69765b95d32cc848a28571f3ad64e"} Nov 03 20:38:52.261258 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:52.261219 2579 generic.go:356] "Generic (PLEG): container finished" podID="c6e5ca9eba1ab368215b5511cd840a25" containerID="8d1cb529c8966a57b48d3f0e7691288895d6376fc4689119efa3515eaaea608a" exitCode=0 Nov 03 20:38:52.261397 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:52.261269 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-175.us-west-1.compute.internal" event={"ID":"c6e5ca9eba1ab368215b5511cd840a25","Type":"ContainerDied","Data":"8d1cb529c8966a57b48d3f0e7691288895d6376fc4689119efa3515eaaea608a"} Nov 03 20:38:53.274521 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:53.274443 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-175.us-west-1.compute.internal" event={"ID":"6873aa06186bc61b65aa0fa7f8c6d9a2","Type":"ContainerStarted","Data":"636a06c7fdd51b61647d612f2378b77fc865760a5467edc80bdf3c1249d19fa1"} Nov 03 20:38:53.280245 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:53.280210 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-175.us-west-1.compute.internal" event={"ID":"c6e5ca9eba1ab368215b5511cd840a25","Type":"ContainerStarted","Data":"3d0abf5deabbd92b2980052e594c39549ce5f0be38e3f75fe387585770e395bb"} Nov 03 20:38:53.283607 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:53.283572 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jvtt8" event={"ID":"3e72f317-1042-40b0-b709-dc3d650754b5","Type":"ContainerStarted","Data":"a896cdeba4cca9f089b6ff38e6fa076d876438587ee1df8edb67056843270a35"} Nov 03 20:38:53.526774 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:53.526695 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-175.us-west-1.compute.internal" podStartSLOduration=7.526674155 podStartE2EDuration="7.526674155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:38:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:38:53.400499359 +0000 UTC m=+8.704834432" watchObservedRunningTime="2026-11-03 20:38:53.526674155 +0000 UTC m=+8.831009232" Nov 03 20:38:53.527049 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:53.527005 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-jvtt8" podStartSLOduration=2.396949227 podStartE2EDuration="7.526997085s" totalImagesPullingTime="5.130047858s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:38:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:38:53.52686354 +0000 UTC m=+8.831198607" watchObservedRunningTime="2026-11-03 20:38:53.526997085 +0000 UTC m=+8.831332158" Nov 03 20:38:53.654691 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:53.654613 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-175.us-west-1.compute.internal" podStartSLOduration=7.654589811 podStartE2EDuration="7.654589811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:38:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:38:53.653793851 +0000 UTC m=+8.958128925" watchObservedRunningTime="2026-11-03 20:38:53.654589811 +0000 UTC m=+8.958924884" Nov 03 20:38:54.042835 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:54.042361 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03985fe9-0dfe-4f9a-97db-bb4c0298be5c-original-pull-secret\") pod \"global-pull-secret-syncer-g4892\" (UID: \"03985fe9-0dfe-4f9a-97db-bb4c0298be5c\") " pod="kube-system/global-pull-secret-syncer-g4892" Nov 03 20:38:54.042835 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:54.042423 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/747345eb-a407-423b-92df-c5d765a7da2d-metrics-certs\") pod \"network-metrics-daemon-xt5qx\" (UID: \"747345eb-a407-423b-92df-c5d765a7da2d\") " pod="openshift-multus/network-metrics-daemon-xt5qx" Nov 03 20:38:54.042835 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:54.042611 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:38:54.042835 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:54.042673 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/747345eb-a407-423b-92df-c5d765a7da2d-metrics-certs podName:747345eb-a407-423b-92df-c5d765a7da2d nodeName:}" failed. No retries permitted until 2026-11-03 20:39:02.042657189 +0000 UTC m=+17.346992255 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/747345eb-a407-423b-92df-c5d765a7da2d-metrics-certs") pod "network-metrics-daemon-xt5qx" (UID: "747345eb-a407-423b-92df-c5d765a7da2d") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:38:54.042835 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:54.042727 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 20:38:54.042835 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:54.042749 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03985fe9-0dfe-4f9a-97db-bb4c0298be5c-original-pull-secret podName:03985fe9-0dfe-4f9a-97db-bb4c0298be5c nodeName:}" failed. No retries permitted until 2026-11-03 20:39:02.042742502 +0000 UTC m=+17.347077565 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/03985fe9-0dfe-4f9a-97db-bb4c0298be5c-original-pull-secret") pod "global-pull-secret-syncer-g4892" (UID: "03985fe9-0dfe-4f9a-97db-bb4c0298be5c") : object "kube-system"/"original-pull-secret" not registered Nov 03 20:38:54.072151 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:54.072116 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-jvtt8" Nov 03 20:38:54.072898 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:54.072870 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-jvtt8" Nov 03 20:38:54.143120 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:54.143080 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rlc8\" (UniqueName: \"kubernetes.io/projected/2dfb2512-b394-4a37-a5ba-a783d4e46c46-kube-api-access-5rlc8\") pod \"network-check-target-hc89p\" (UID: \"2dfb2512-b394-4a37-a5ba-a783d4e46c46\") " pod="openshift-network-diagnostics/network-check-target-hc89p" Nov 03 20:38:54.143302 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:54.143288 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 20:38:54.143377 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:54.143311 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 20:38:54.143377 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:54.143325 2579 projected.go:196] Error preparing data for projected volume kube-api-access-5rlc8 for pod openshift-network-diagnostics/network-check-target-hc89p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:38:54.143452 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:54.143389 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2dfb2512-b394-4a37-a5ba-a783d4e46c46-kube-api-access-5rlc8 podName:2dfb2512-b394-4a37-a5ba-a783d4e46c46 nodeName:}" failed. No retries permitted until 2026-11-03 20:39:02.143370517 +0000 UTC m=+17.447705581 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-5rlc8" (UniqueName: "kubernetes.io/projected/2dfb2512-b394-4a37-a5ba-a783d4e46c46-kube-api-access-5rlc8") pod "network-check-target-hc89p" (UID: "2dfb2512-b394-4a37-a5ba-a783d4e46c46") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:38:54.181385 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:54.181347 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xt5qx" Nov 03 20:38:54.181608 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:54.181529 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xt5qx" podUID="747345eb-a407-423b-92df-c5d765a7da2d" Nov 03 20:38:54.181676 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:54.181611 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hc89p" Nov 03 20:38:54.181728 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:54.181689 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hc89p" podUID="2dfb2512-b394-4a37-a5ba-a783d4e46c46" Nov 03 20:38:54.181789 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:54.181751 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g4892" Nov 03 20:38:54.181855 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:54.181825 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g4892" podUID="03985fe9-0dfe-4f9a-97db-bb4c0298be5c" Nov 03 20:38:54.288629 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:54.288590 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq7n8" event={"ID":"bafd96d7-6982-4c93-81fc-d61643002fd1","Type":"ContainerStarted","Data":"605a20f06fb60728a6c0e7467cf33ecd0cbc29158872c7b01782034944c449bb"} Nov 03 20:38:54.292104 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:54.292063 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-p6hwx" event={"ID":"7ced550a-b3cb-4605-b8dc-aab3b55804b4","Type":"ContainerStarted","Data":"26faab6dbe3a4305bdf9aaa8ea78b7fc05df20e26469f88014c432cdb0892504"} Nov 03 20:38:55.256385 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:55.256302 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-jvtt8" Nov 03 20:38:55.257776 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:55.257536 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jvtt8" Nov 03 20:38:55.385981 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:55.384595 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-p6hwx" podStartSLOduration=2.8888868949999997 podStartE2EDuration="9.384571303s" totalImagesPullingTime="6.495684408s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:38:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:38:54.426750313 +0000 UTC m=+9.731085406" watchObservedRunningTime="2026-11-03 20:38:55.384571303 +0000 UTC m=+10.688906371" Nov 03 20:38:56.180736 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:56.180635 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g4892" Nov 03 20:38:56.180898 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:56.180774 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g4892" podUID="03985fe9-0dfe-4f9a-97db-bb4c0298be5c" Nov 03 20:38:56.180898 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:56.180844 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xt5qx" Nov 03 20:38:56.180995 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:56.180947 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xt5qx" podUID="747345eb-a407-423b-92df-c5d765a7da2d" Nov 03 20:38:56.181052 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:56.180999 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hc89p" Nov 03 20:38:56.181111 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:56.181071 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hc89p" podUID="2dfb2512-b394-4a37-a5ba-a783d4e46c46" Nov 03 20:38:56.254370 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:56.254323 2579 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 20:38:56.300098 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:56.300067 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4k6r2" event={"ID":"4a5df9f4-aaaa-4441-a2e0-28744e197828","Type":"ContainerStarted","Data":"43edb169685dfdff8e7176448bd48506ea8ac96f3e8a2f7c518dbeab2757ebea"} Nov 03 20:38:56.304232 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:56.303989 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq7n8" event={"ID":"bafd96d7-6982-4c93-81fc-d61643002fd1","Type":"ContainerStarted","Data":"79ad779bdbb1a1792e849c112b38891c2fd8c10a944a3c565dc2663708294c0e"} Nov 03 20:38:57.138978 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:57.138603 2579 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T20:38:56.254349621Z","UUID":"2ea12c13-3add-4c28-add3-292b50367c14","Handler":null,"Name":"","Endpoint":""} Nov 03 20:38:57.140499 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:57.140475 2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 20:38:57.140624 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:57.140509 2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 20:38:57.309515 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:57.309473 2579 generic.go:356] "Generic (PLEG): container finished" podID="2f1f5ba6-ccc5-4f0d-a135-743747ce667a" containerID="7c9155ccdf102f26a2ebe2477003e400f826a3f09a3af906af3c47c013d53db3" exitCode=0 Nov 03 20:38:57.309714 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:57.309585 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-shzwn" event={"ID":"2f1f5ba6-ccc5-4f0d-a135-743747ce667a","Type":"ContainerDied","Data":"7c9155ccdf102f26a2ebe2477003e400f826a3f09a3af906af3c47c013d53db3"} Nov 03 20:38:57.313576 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:57.313548 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gkm7b" event={"ID":"cf15bf7c-07c6-4d82-987b-aa4a332d5ca9","Type":"ContainerStarted","Data":"d9b06cff2a9aa48bf2798ae6af0c87f161f5472483f15d68621cf7cb29021728"} Nov 03 20:38:57.443006 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:57.442940 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-4k6r2" podStartSLOduration=2.03516223 podStartE2EDuration="11.442921104s" totalImagesPullingTime="9.407758874s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:38:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:38:56.432261033 +0000 UTC m=+11.736596139" watchObservedRunningTime="2026-11-03 20:38:57.442921104 +0000 UTC m=+12.747256177" Nov 03 20:38:57.574876 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:57.574760 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-gkm7b" podStartSLOduration=2.228867079 podStartE2EDuration="11.574737578s" totalImagesPullingTime="9.345870499s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:38:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:38:57.574496162 +0000 UTC m=+12.878831235" watchObservedRunningTime="2026-11-03 20:38:57.574737578 +0000 UTC m=+12.879072652" Nov 03 20:38:58.180453 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:58.180401 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xt5qx" Nov 03 20:38:58.180900 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:58.180401 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g4892" Nov 03 20:38:58.180900 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:58.180564 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hc89p" Nov 03 20:38:58.180900 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:58.180567 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xt5qx" podUID="747345eb-a407-423b-92df-c5d765a7da2d" Nov 03 20:38:58.180900 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:58.180661 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g4892" podUID="03985fe9-0dfe-4f9a-97db-bb4c0298be5c" Nov 03 20:38:58.180900 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:38:58.180730 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hc89p" podUID="2dfb2512-b394-4a37-a5ba-a783d4e46c46" Nov 03 20:38:58.317915 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:58.317834 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lvcbj" event={"ID":"6e89ebac-2191-4acc-b4d4-ff326b3b74ac","Type":"ContainerStarted","Data":"e1aa40d39f3651e9ca6eadec28030596ca2797fd5d8aa971f61748f2aef42db0"} Nov 03 20:38:58.435963 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:58.435675 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-lvcbj" podStartSLOduration=2.7612124749999998 podStartE2EDuration="12.435654597s" totalImagesPullingTime="9.674442122s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:38:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:38:58.434794297 +0000 UTC m=+13.739129365" watchObservedRunningTime="2026-11-03 20:38:58.435654597 +0000 UTC m=+13.739989669" Nov 03 20:38:59.321033 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:59.320948 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" event={"ID":"6d81822a-bda4-4f5e-a5f4-05e1606f22bf","Type":"ContainerStarted","Data":"f0d6ea366d92ac95fd2e14a6eb49673a2bfa2e25b3aa368eb08f24d06b4cd65b"} Nov 03 20:38:59.322995 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:59.322974 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq7n8" event={"ID":"bafd96d7-6982-4c93-81fc-d61643002fd1","Type":"ContainerStarted","Data":"7ab567de2032ec22bf6b68f948f8d3735e32dbea80305af39237a5f8476643fe"} Nov 03 20:38:59.325031 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:59.325007 2579 generic.go:356] "Generic (PLEG): container finished" podID="2f1f5ba6-ccc5-4f0d-a135-743747ce667a" containerID="d6bae224e7649b543b1c87f0ac71981b8487450781ea430ee15dbc3f1f43145d" exitCode=0 Nov 03 20:38:59.325127 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:59.325078 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-shzwn" event={"ID":"2f1f5ba6-ccc5-4f0d-a135-743747ce667a","Type":"ContainerDied","Data":"d6bae224e7649b543b1c87f0ac71981b8487450781ea430ee15dbc3f1f43145d"} Nov 03 20:38:59.441735 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:59.441681 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" podStartSLOduration=2.002534891 podStartE2EDuration="13.441663217s" totalImagesPullingTime="11.439128326s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:38:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:38:59.440418693 +0000 UTC m=+14.744753762" watchObservedRunningTime="2026-11-03 20:38:59.441663217 +0000 UTC m=+14.745998287" Nov 03 20:38:59.685580 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:38:59.685482 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq7n8" podStartSLOduration=2.576619659 podStartE2EDuration="13.685465286s" totalImagesPullingTime="11.108845627s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:38:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:38:59.684775734 +0000 UTC m=+14.989110806" watchObservedRunningTime="2026-11-03 20:38:59.685465286 +0000 UTC m=+14.989800355" Nov 03 20:39:00.181188 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:00.181104 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g4892" Nov 03 20:39:00.181188 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:00.181158 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xt5qx" Nov 03 20:39:00.181188 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:00.181110 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hc89p" Nov 03 20:39:00.181427 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:39:00.181233 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g4892" podUID="03985fe9-0dfe-4f9a-97db-bb4c0298be5c" Nov 03 20:39:00.181427 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:39:00.181317 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hc89p" podUID="2dfb2512-b394-4a37-a5ba-a783d4e46c46" Nov 03 20:39:00.181427 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:39:00.181401 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xt5qx" podUID="747345eb-a407-423b-92df-c5d765a7da2d" Nov 03 20:39:01.332156 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:01.332122 2579 generic.go:356] "Generic (PLEG): container finished" podID="2f1f5ba6-ccc5-4f0d-a135-743747ce667a" containerID="566bae121881cf64b970d56bfe6f11e3ee068e6e1aad25b901da6a04544dad07" exitCode=0 Nov 03 20:39:01.332958 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:01.332172 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-shzwn" event={"ID":"2f1f5ba6-ccc5-4f0d-a135-743747ce667a","Type":"ContainerDied","Data":"566bae121881cf64b970d56bfe6f11e3ee068e6e1aad25b901da6a04544dad07"} Nov 03 20:39:02.111846 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:02.111803 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03985fe9-0dfe-4f9a-97db-bb4c0298be5c-original-pull-secret\") pod \"global-pull-secret-syncer-g4892\" (UID: \"03985fe9-0dfe-4f9a-97db-bb4c0298be5c\") " pod="kube-system/global-pull-secret-syncer-g4892" Nov 03 20:39:02.112041 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:02.111865 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/747345eb-a407-423b-92df-c5d765a7da2d-metrics-certs\") pod \"network-metrics-daemon-xt5qx\" (UID: \"747345eb-a407-423b-92df-c5d765a7da2d\") " pod="openshift-multus/network-metrics-daemon-xt5qx" Nov 03 20:39:02.112041 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:39:02.111975 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:39:02.112041 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:39:02.111981 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 20:39:02.112041 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:39:02.112036 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/747345eb-a407-423b-92df-c5d765a7da2d-metrics-certs podName:747345eb-a407-423b-92df-c5d765a7da2d nodeName:}" failed. No retries permitted until 2026-11-03 20:39:18.112019198 +0000 UTC m=+33.416354265 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/747345eb-a407-423b-92df-c5d765a7da2d-metrics-certs") pod "network-metrics-daemon-xt5qx" (UID: "747345eb-a407-423b-92df-c5d765a7da2d") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:39:02.112211 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:39:02.112119 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03985fe9-0dfe-4f9a-97db-bb4c0298be5c-original-pull-secret podName:03985fe9-0dfe-4f9a-97db-bb4c0298be5c nodeName:}" failed. No retries permitted until 2026-11-03 20:39:18.112107812 +0000 UTC m=+33.416442863 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/03985fe9-0dfe-4f9a-97db-bb4c0298be5c-original-pull-secret") pod "global-pull-secret-syncer-g4892" (UID: "03985fe9-0dfe-4f9a-97db-bb4c0298be5c") : object "kube-system"/"original-pull-secret" not registered Nov 03 20:39:02.181327 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:02.181294 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g4892" Nov 03 20:39:02.181506 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:02.181308 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xt5qx" Nov 03 20:39:02.181506 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:39:02.181412 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g4892" podUID="03985fe9-0dfe-4f9a-97db-bb4c0298be5c" Nov 03 20:39:02.181506 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:02.181445 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hc89p" Nov 03 20:39:02.181633 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:39:02.181547 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xt5qx" podUID="747345eb-a407-423b-92df-c5d765a7da2d" Nov 03 20:39:02.181666 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:39:02.181638 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hc89p" podUID="2dfb2512-b394-4a37-a5ba-a783d4e46c46" Nov 03 20:39:02.212636 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:02.212599 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rlc8\" (UniqueName: \"kubernetes.io/projected/2dfb2512-b394-4a37-a5ba-a783d4e46c46-kube-api-access-5rlc8\") pod \"network-check-target-hc89p\" (UID: \"2dfb2512-b394-4a37-a5ba-a783d4e46c46\") " pod="openshift-network-diagnostics/network-check-target-hc89p" Nov 03 20:39:02.212808 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:39:02.212746 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 20:39:02.212808 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:39:02.212761 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 20:39:02.212808 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:39:02.212772 2579 projected.go:196] Error preparing data for projected volume kube-api-access-5rlc8 for pod openshift-network-diagnostics/network-check-target-hc89p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:39:02.212934 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:39:02.212829 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2dfb2512-b394-4a37-a5ba-a783d4e46c46-kube-api-access-5rlc8 podName:2dfb2512-b394-4a37-a5ba-a783d4e46c46 nodeName:}" failed. No retries permitted until 2026-11-03 20:39:18.212815979 +0000 UTC m=+33.517151039 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-5rlc8" (UniqueName: "kubernetes.io/projected/2dfb2512-b394-4a37-a5ba-a783d4e46c46-kube-api-access-5rlc8") pod "network-check-target-hc89p" (UID: "2dfb2512-b394-4a37-a5ba-a783d4e46c46") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:39:04.180728 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:04.180363 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g4892" Nov 03 20:39:04.181132 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:04.180450 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hc89p" Nov 03 20:39:04.181132 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:39:04.180759 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g4892" podUID="03985fe9-0dfe-4f9a-97db-bb4c0298be5c" Nov 03 20:39:04.181132 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:04.180458 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xt5qx" Nov 03 20:39:04.181132 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:39:04.180835 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hc89p" podUID="2dfb2512-b394-4a37-a5ba-a783d4e46c46" Nov 03 20:39:04.181132 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:39:04.180907 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xt5qx" podUID="747345eb-a407-423b-92df-c5d765a7da2d" Nov 03 20:39:04.341684 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:04.341650 2579 generic.go:356] "Generic (PLEG): container finished" podID="2f1f5ba6-ccc5-4f0d-a135-743747ce667a" containerID="04a3b107559ce81fa0c42ff33e1881cd58b33943b5e722ae931ab3d9c15fd440" exitCode=0 Nov 03 20:39:04.341915 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:04.341722 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-shzwn" event={"ID":"2f1f5ba6-ccc5-4f0d-a135-743747ce667a","Type":"ContainerDied","Data":"04a3b107559ce81fa0c42ff33e1881cd58b33943b5e722ae931ab3d9c15fd440"} Nov 03 20:39:04.344754 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:04.344732 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" event={"ID":"12658495-b5af-4c01-9c72-19b1173f7529","Type":"ContainerStarted","Data":"c98760e4b60bfe720865b738e5d0938569e5f9de136c2dcf7a2b50d8ecb34684"} Nov 03 20:39:04.344824 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:04.344796 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" event={"ID":"12658495-b5af-4c01-9c72-19b1173f7529","Type":"ContainerStarted","Data":"58071ee629b00d3a61b603a982a6b1a05e1aaf0353e7d017277cd3f73fc197a0"} Nov 03 20:39:04.344893 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:04.344830 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" event={"ID":"12658495-b5af-4c01-9c72-19b1173f7529","Type":"ContainerStarted","Data":"338f6bc37deea8a57c96e108cb73679b8783003628fc71a9929fd2de1c56dbf7"} Nov 03 20:39:04.344893 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:04.344858 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" event={"ID":"12658495-b5af-4c01-9c72-19b1173f7529","Type":"ContainerStarted","Data":"877494d34de917af413435e7ce57ebf8fb82bc893d667eb4e4478ab40de44a0a"} Nov 03 20:39:04.344893 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:04.344891 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" event={"ID":"12658495-b5af-4c01-9c72-19b1173f7529","Type":"ContainerStarted","Data":"c41f38e058c5b1aa97c96d4d00b5260135c4a169a4256d15a179f28080125c99"} Nov 03 20:39:04.345028 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:04.344917 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" event={"ID":"12658495-b5af-4c01-9c72-19b1173f7529","Type":"ContainerStarted","Data":"789bc4d217e8565cdf87a0b42a19641a9adb1c1414452fc042aa41326675c81f"} Nov 03 20:39:05.349990 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:05.349962 2579 generic.go:356] "Generic (PLEG): container finished" podID="2f1f5ba6-ccc5-4f0d-a135-743747ce667a" containerID="9901fb3edf2819a2fc642ce42002a80d440bb14c3b735f61c0b63307f8d28756" exitCode=0 Nov 03 20:39:05.350368 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:05.350021 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-shzwn" event={"ID":"2f1f5ba6-ccc5-4f0d-a135-743747ce667a","Type":"ContainerDied","Data":"9901fb3edf2819a2fc642ce42002a80d440bb14c3b735f61c0b63307f8d28756"} Nov 03 20:39:06.181421 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:06.181377 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xt5qx" Nov 03 20:39:06.181594 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:06.181476 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hc89p" Nov 03 20:39:06.181594 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:06.181492 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g4892" Nov 03 20:39:06.181594 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:39:06.181575 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hc89p" podUID="2dfb2512-b394-4a37-a5ba-a783d4e46c46" Nov 03 20:39:06.181679 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:39:06.181634 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xt5qx" podUID="747345eb-a407-423b-92df-c5d765a7da2d" Nov 03 20:39:06.181738 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:39:06.181715 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g4892" podUID="03985fe9-0dfe-4f9a-97db-bb4c0298be5c" Nov 03 20:39:06.355072 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:06.354994 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" event={"ID":"12658495-b5af-4c01-9c72-19b1173f7529","Type":"ContainerStarted","Data":"acee7574cbcd02bf5d0e6e69b7a693685641f47d603a7941113ea7dc545f0979"} Nov 03 20:39:06.358213 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:06.358185 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-shzwn" event={"ID":"2f1f5ba6-ccc5-4f0d-a135-743747ce667a","Type":"ContainerStarted","Data":"fc6bd959d26ba452c78568bf815e89f4a00a4bc6b8243716393723c549e5cebb"} Nov 03 20:39:06.481895 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:06.481844 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-shzwn" podStartSLOduration=6.737880586 podStartE2EDuration="20.481829398s" totalImagesPullingTime="13.743948812s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:38:46 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:39:06.48132425 +0000 UTC m=+21.785659318" watchObservedRunningTime="2026-11-03 20:39:06.481829398 +0000 UTC m=+21.786164467" Nov 03 20:39:08.181008 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:08.180982 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g4892" Nov 03 20:39:08.181523 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:08.181010 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xt5qx" Nov 03 20:39:08.181523 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:08.181013 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hc89p" Nov 03 20:39:08.181523 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:39:08.181108 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g4892" podUID="03985fe9-0dfe-4f9a-97db-bb4c0298be5c" Nov 03 20:39:08.181523 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:39:08.181166 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xt5qx" podUID="747345eb-a407-423b-92df-c5d765a7da2d" Nov 03 20:39:08.181523 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:39:08.181225 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hc89p" podUID="2dfb2512-b394-4a37-a5ba-a783d4e46c46" Nov 03 20:39:08.366426 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:08.366336 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" event={"ID":"12658495-b5af-4c01-9c72-19b1173f7529","Type":"ContainerStarted","Data":"95f89988d4ab07e151e1678fa6ebaadb0c18d4cf3c7379612e922e6ee86a1424"} Nov 03 20:39:08.366655 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:08.366638 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:39:08.382941 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:08.382914 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:39:08.648304 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:08.648209 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" podStartSLOduration=5.7565271 podStartE2EDuration="22.648196295s" totalImagesPullingTime="16.891669195s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:38:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:39:08.597200006 +0000 UTC m=+23.901535075" watchObservedRunningTime="2026-11-03 20:39:08.648196295 +0000 UTC m=+23.952531364" Nov 03 20:39:09.370760 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:09.370725 2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 20:39:09.371317 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:09.371235 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:39:09.388506 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:09.388480 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:39:10.180804 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:10.180766 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hc89p" Nov 03 20:39:10.181008 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:10.180841 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xt5qx" Nov 03 20:39:10.181008 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:10.180862 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g4892" Nov 03 20:39:10.181008 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:39:10.180961 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hc89p" podUID="2dfb2512-b394-4a37-a5ba-a783d4e46c46" Nov 03 20:39:10.181176 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:39:10.181068 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xt5qx" podUID="747345eb-a407-423b-92df-c5d765a7da2d" Nov 03 20:39:10.181176 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:39:10.181132 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g4892" podUID="03985fe9-0dfe-4f9a-97db-bb4c0298be5c" Nov 03 20:39:10.372651 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:10.372626 2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 20:39:10.896620 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:10.896327 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hc89p"] Nov 03 20:39:10.896782 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:10.896634 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hc89p" Nov 03 20:39:10.896782 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:39:10.896726 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hc89p" podUID="2dfb2512-b394-4a37-a5ba-a783d4e46c46" Nov 03 20:39:10.896943 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:10.896918 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-g4892"] Nov 03 20:39:10.897023 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:10.897012 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g4892" Nov 03 20:39:10.897108 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:39:10.897093 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g4892" podUID="03985fe9-0dfe-4f9a-97db-bb4c0298be5c" Nov 03 20:39:10.899147 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:10.899125 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xt5qx"] Nov 03 20:39:10.899276 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:10.899214 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xt5qx" Nov 03 20:39:10.899344 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:39:10.899295 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xt5qx" podUID="747345eb-a407-423b-92df-c5d765a7da2d" Nov 03 20:39:11.374671 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:11.374594 2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 20:39:12.180716 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:12.180685 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hc89p" Nov 03 20:39:12.180716 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:12.180718 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g4892" Nov 03 20:39:12.181006 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:39:12.180783 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hc89p" podUID="2dfb2512-b394-4a37-a5ba-a783d4e46c46" Nov 03 20:39:12.181006 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:39:12.180898 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g4892" podUID="03985fe9-0dfe-4f9a-97db-bb4c0298be5c" Nov 03 20:39:13.089966 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:13.089926 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:39:13.090385 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:13.090118 2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 20:39:13.101556 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:13.101510 2579 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" podUID="12658495-b5af-4c01-9c72-19b1173f7529" containerName="ovnkube-controller" probeResult="failure" output="" Nov 03 20:39:13.110984 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:13.110953 2579 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" podUID="12658495-b5af-4c01-9c72-19b1173f7529" containerName="ovnkube-controller" probeResult="failure" output="" Nov 03 20:39:13.181042 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:13.180955 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xt5qx" Nov 03 20:39:13.181190 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:39:13.181074 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xt5qx" podUID="747345eb-a407-423b-92df-c5d765a7da2d" Nov 03 20:39:14.180836 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:14.180809 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g4892" Nov 03 20:39:14.181239 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:14.180807 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hc89p" Nov 03 20:39:14.181239 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:39:14.180903 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g4892" podUID="03985fe9-0dfe-4f9a-97db-bb4c0298be5c" Nov 03 20:39:14.181239 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:39:14.180977 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hc89p" podUID="2dfb2512-b394-4a37-a5ba-a783d4e46c46" Nov 03 20:39:15.185397 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:15.185362 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xt5qx" Nov 03 20:39:15.185868 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:39:15.185498 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xt5qx" podUID="747345eb-a407-423b-92df-c5d765a7da2d" Nov 03 20:39:15.988069 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:15.988041 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-175.us-west-1.compute.internal" event="NodeReady" Nov 03 20:39:15.988225 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:15.988173 2579 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 20:39:16.181228 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:16.181195 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g4892" Nov 03 20:39:16.181228 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:16.181223 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hc89p" Nov 03 20:39:16.236604 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:16.236576 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mtgd7\"" Nov 03 20:39:16.236972 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:16.236606 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-zvv9n\"" Nov 03 20:39:16.236972 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:16.236722 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 20:39:16.236972 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:16.236768 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 20:39:16.236972 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:16.236793 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 20:39:17.180910 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:17.180872 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xt5qx" Nov 03 20:39:17.312501 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:17.312464 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 20:39:17.312923 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:17.312578 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-j2mnh\"" Nov 03 20:39:18.131645 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:18.131610 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/747345eb-a407-423b-92df-c5d765a7da2d-metrics-certs\") pod \"network-metrics-daemon-xt5qx\" (UID: \"747345eb-a407-423b-92df-c5d765a7da2d\") " pod="openshift-multus/network-metrics-daemon-xt5qx" Nov 03 20:39:18.131835 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:18.131671 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03985fe9-0dfe-4f9a-97db-bb4c0298be5c-original-pull-secret\") pod \"global-pull-secret-syncer-g4892\" (UID: \"03985fe9-0dfe-4f9a-97db-bb4c0298be5c\") " pod="kube-system/global-pull-secret-syncer-g4892" Nov 03 20:39:18.131835 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:39:18.131757 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 20:39:18.131835 ip-10-0-128-175 kubenswrapper[2579]: E1103 20:39:18.131818 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/747345eb-a407-423b-92df-c5d765a7da2d-metrics-certs podName:747345eb-a407-423b-92df-c5d765a7da2d nodeName:}" failed. No retries permitted until 2026-11-03 20:39:50.131803866 +0000 UTC m=+65.436138929 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/747345eb-a407-423b-92df-c5d765a7da2d-metrics-certs") pod "network-metrics-daemon-xt5qx" (UID: "747345eb-a407-423b-92df-c5d765a7da2d") : secret "metrics-daemon-secret" not found Nov 03 20:39:18.135706 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:18.135685 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03985fe9-0dfe-4f9a-97db-bb4c0298be5c-original-pull-secret\") pod \"global-pull-secret-syncer-g4892\" (UID: \"03985fe9-0dfe-4f9a-97db-bb4c0298be5c\") " pod="kube-system/global-pull-secret-syncer-g4892" Nov 03 20:39:18.232904 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:18.232863 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rlc8\" (UniqueName: \"kubernetes.io/projected/2dfb2512-b394-4a37-a5ba-a783d4e46c46-kube-api-access-5rlc8\") pod \"network-check-target-hc89p\" (UID: \"2dfb2512-b394-4a37-a5ba-a783d4e46c46\") " pod="openshift-network-diagnostics/network-check-target-hc89p" Nov 03 20:39:18.235446 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:18.235403 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rlc8\" (UniqueName: \"kubernetes.io/projected/2dfb2512-b394-4a37-a5ba-a783d4e46c46-kube-api-access-5rlc8\") pod \"network-check-target-hc89p\" (UID: \"2dfb2512-b394-4a37-a5ba-a783d4e46c46\") " pod="openshift-network-diagnostics/network-check-target-hc89p" Nov 03 20:39:18.296407 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:18.296366 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g4892" Nov 03 20:39:18.298538 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:18.298519 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hc89p" Nov 03 20:39:18.791140 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:18.791060 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-g4892"] Nov 03 20:39:18.793850 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:18.793825 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hc89p"] Nov 03 20:39:18.793981 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:39:18.793890 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03985fe9_0dfe_4f9a_97db_bb4c0298be5c.slice/crio-1ff43da0fa130ddee036756b0058aaa333d6b988c333f92a80145ca74248dd06 WatchSource:0}: Error finding container 1ff43da0fa130ddee036756b0058aaa333d6b988c333f92a80145ca74248dd06: Status 404 returned error can't find the container with id 1ff43da0fa130ddee036756b0058aaa333d6b988c333f92a80145ca74248dd06 Nov 03 20:39:18.833278 ip-10-0-128-175 kubenswrapper[2579]: W1103 20:39:18.833252 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dfb2512_b394_4a37_a5ba_a783d4e46c46.slice/crio-c43e801572ab4ba97c6f192f69c89cab85583a68c7ae5d61899f09c02737f8b4 WatchSource:0}: Error finding container c43e801572ab4ba97c6f192f69c89cab85583a68c7ae5d61899f09c02737f8b4: Status 404 returned error can't find the container with id c43e801572ab4ba97c6f192f69c89cab85583a68c7ae5d61899f09c02737f8b4 Nov 03 20:39:19.157645 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:19.157624 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gkm7b_cf15bf7c-07c6-4d82-987b-aa4a332d5ca9/dns-node-resolver/0.log" Nov 03 20:39:19.393405 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:19.393368 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hc89p" event={"ID":"2dfb2512-b394-4a37-a5ba-a783d4e46c46","Type":"ContainerStarted","Data":"c43e801572ab4ba97c6f192f69c89cab85583a68c7ae5d61899f09c02737f8b4"} Nov 03 20:39:19.394544 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:19.394517 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-g4892" event={"ID":"03985fe9-0dfe-4f9a-97db-bb4c0298be5c","Type":"ContainerStarted","Data":"1ff43da0fa130ddee036756b0058aaa333d6b988c333f92a80145ca74248dd06"} Nov 03 20:39:20.174840 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:20.174807 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-p6hwx_7ced550a-b3cb-4605-b8dc-aab3b55804b4/node-ca/0.log" Nov 03 20:39:23.406801 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:23.406767 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hc89p" event={"ID":"2dfb2512-b394-4a37-a5ba-a783d4e46c46","Type":"ContainerStarted","Data":"4004b05e802dc2a56ad2bf2de4a0f7f95e279b0b37475ec6e93ece4c1a75de94"} Nov 03 20:39:23.407293 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:23.406934 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-hc89p" Nov 03 20:39:23.524788 ip-10-0-128-175 kubenswrapper[2579]: I1103 20:39:23.524717 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-hc89p" podStartSLOduration=33.489200099 podStartE2EDuration="37.524700008s" totalImagesPullingTime="4.035499909s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:38:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:39:23.523804448 +0000 UTC m=+38.828139531" watchObservedRunningTime="2026-11-03 20:39:23.524700008 +0000 UTC m=+38.829035078" Nov 03 20:39:24.708677 ip-10-0-128-175 systemd[1]: Stopping Kubernetes Kubelet... Nov 03 20:39:24.723919 ip-10-0-128-175 systemd[1]: kubelet.service: Deactivated successfully. Nov 03 20:39:24.724116 ip-10-0-128-175 systemd[1]: Stopped Kubernetes Kubelet. Nov 03 20:39:24.724171 ip-10-0-128-175 systemd[1]: kubelet.service: Consumed 2.219s CPU time. Nov 03 20:39:24.743147 ip-10-0-128-175 systemd[1]: Starting Kubernetes Kubelet... Nov 03 20:39:24.817306 ip-10-0-128-175 kubenswrapper[4305]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 20:39:24.817306 ip-10-0-128-175 kubenswrapper[4305]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 20:39:24.817306 ip-10-0-128-175 kubenswrapper[4305]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 20:39:24.817306 ip-10-0-128-175 kubenswrapper[4305]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 20:39:24.819469 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819365 4305 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:39:24.819469 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819470 4305 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:39:24.819469 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819473 4305 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:39:24.819567 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819476 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:39:24.819567 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819479 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:39:24.819567 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819484 4305 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:39:24.819567 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819488 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:39:24.819567 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819491 4305 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:39:24.819567 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819494 4305 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:39:24.819567 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819497 4305 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:39:24.819567 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819500 4305 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:39:24.819567 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819502 4305 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:39:24.819567 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819505 4305 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:39:24.819567 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819507 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:39:24.819567 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819510 4305 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:39:24.819567 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819512 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:39:24.819567 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819515 4305 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:39:24.819567 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819518 4305 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:39:24.819567 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819520 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:39:24.819567 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819523 4305 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:39:24.819567 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819525 4305 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:39:24.819567 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819528 4305 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:39:24.820051 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819530 4305 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:39:24.820051 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819533 4305 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:39:24.820051 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819536 4305 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:39:24.820051 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819538 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:39:24.820051 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819540 4305 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:39:24.820051 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819551 4305 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:39:24.820051 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819555 4305 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:39:24.820051 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819557 4305 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:39:24.820051 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819560 4305 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:39:24.820051 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819563 4305 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:39:24.820051 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819566 4305 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:39:24.820051 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819568 4305 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:39:24.820051 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819571 4305 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:39:24.820051 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819574 4305 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:39:24.820051 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819577 4305 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:39:24.820051 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819579 4305 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:39:24.820051 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819582 4305 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:39:24.820051 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819585 4305 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:39:24.820051 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819587 4305 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:39:24.820544 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819590 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:39:24.820544 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819593 4305 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:39:24.820544 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819595 4305 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:39:24.820544 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819598 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:39:24.820544 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819600 4305 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:39:24.820544 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819603 4305 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:39:24.820544 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819605 4305 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:39:24.820544 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819608 4305 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:39:24.820544 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819611 4305 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:39:24.820544 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819613 4305 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:39:24.820544 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819615 4305 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:39:24.820544 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819618 4305 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:39:24.820544 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819621 4305 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:39:24.820544 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819623 4305 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:39:24.820544 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819626 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:39:24.820544 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819628 4305 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:39:24.820544 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819631 4305 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:39:24.820544 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819634 4305 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:39:24.820544 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819637 4305 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:39:24.821013 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819641 4305 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:39:24.821013 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819645 4305 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:39:24.821013 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819647 4305 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:39:24.821013 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819650 4305 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:39:24.821013 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819652 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:39:24.821013 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819656 4305 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:39:24.821013 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819658 4305 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:39:24.821013 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819661 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:39:24.821013 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819664 4305 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:39:24.821013 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819666 4305 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:39:24.821013 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819668 4305 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:39:24.821013 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819671 4305 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:39:24.821013 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819673 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:39:24.821013 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819676 4305 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:39:24.821013 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819678 4305 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:39:24.821013 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819681 4305 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:39:24.821013 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819683 4305 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:39:24.821013 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819686 4305 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:39:24.821013 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819689 4305 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:39:24.821013 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819691 4305 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:39:24.821559 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819694 4305 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:39:24.821559 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819696 4305 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:39:24.821559 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819699 4305 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:39:24.821559 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819702 4305 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:39:24.821559 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819705 4305 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:39:24.821559 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819707 4305 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:39:24.821559 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819710 4305 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:39:24.821559 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819712 4305 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:39:24.821559 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819714 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:39:24.821559 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819717 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:39:24.821559 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819720 4305 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:39:24.821559 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819722 4305 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:39:24.821559 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819727 4305 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:39:24.821559 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819737 4305 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:39:24.821559 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819740 4305 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:39:24.821559 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819743 4305 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:39:24.821559 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819745 4305 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:39:24.821559 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819748 4305 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:39:24.821559 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819751 4305 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:39:24.822069 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819753 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:39:24.822069 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819755 4305 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:39:24.822069 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819758 4305 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:39:24.822069 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819760 4305 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:39:24.822069 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819763 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:39:24.822069 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819765 4305 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:39:24.822069 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.819768 4305 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:39:24.822069 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820545 4305 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:39:24.822069 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820553 4305 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:39:24.822069 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820555 4305 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:39:24.822069 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820558 4305 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:39:24.822069 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820563 4305 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:39:24.822069 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820567 4305 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:39:24.822069 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820570 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:39:24.822069 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820574 4305 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:39:24.822069 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820578 4305 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:39:24.822069 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820581 4305 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:39:24.822069 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820584 4305 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:39:24.822069 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820586 4305 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:39:24.822559 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820589 4305 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:39:24.822559 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820591 4305 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:39:24.822559 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820594 4305 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:39:24.822559 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820596 4305 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:39:24.822559 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820600 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:39:24.822559 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820602 4305 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:39:24.822559 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820605 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:39:24.822559 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820608 4305 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:39:24.822559 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820612 4305 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:39:24.822559 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820614 4305 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:39:24.822559 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820617 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:39:24.822559 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820620 4305 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:39:24.822559 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820623 4305 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:39:24.822559 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820626 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:39:24.822559 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820628 4305 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:39:24.822559 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820631 4305 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:39:24.822559 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820633 4305 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:39:24.822559 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820636 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:39:24.822559 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820639 4305 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:39:24.822559 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820641 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:39:24.823056 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820644 4305 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:39:24.823056 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820655 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:39:24.823056 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820658 4305 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:39:24.823056 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820661 4305 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:39:24.823056 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820663 4305 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:39:24.823056 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820666 4305 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:39:24.823056 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820668 4305 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:39:24.823056 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820671 4305 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:39:24.823056 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820673 4305 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:39:24.823056 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820676 4305 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:39:24.823056 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820679 4305 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:39:24.823056 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820681 4305 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:39:24.823056 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820684 4305 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:39:24.823056 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820686 4305 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:39:24.823056 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820689 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:39:24.823056 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820691 4305 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:39:24.823056 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820694 4305 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:39:24.823056 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820696 4305 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:39:24.823056 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820699 4305 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:39:24.823538 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820702 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:39:24.823538 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820704 4305 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:39:24.823538 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820707 4305 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:39:24.823538 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820710 4305 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:39:24.823538 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820713 4305 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:39:24.823538 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820716 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:39:24.823538 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820719 4305 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:39:24.823538 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820721 4305 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:39:24.823538 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820724 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:39:24.823538 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820727 4305 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:39:24.823538 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820729 4305 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:39:24.823538 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820733 4305 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:39:24.823538 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820735 4305 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:39:24.823538 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820738 4305 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:39:24.823538 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820740 4305 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:39:24.823538 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820743 4305 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:39:24.823538 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820745 4305 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:39:24.823538 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820748 4305 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:39:24.823538 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820750 4305 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:39:24.823538 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820753 4305 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:39:24.824089 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820755 4305 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:39:24.824089 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820758 4305 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:39:24.824089 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820760 4305 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:39:24.824089 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820763 4305 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:39:24.824089 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820765 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:39:24.824089 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820767 4305 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:39:24.824089 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820770 4305 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:39:24.824089 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820772 4305 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:39:24.824089 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820774 4305 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:39:24.824089 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820782 4305 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:39:24.824089 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820785 4305 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:39:24.824089 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820787 4305 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:39:24.824089 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820790 4305 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:39:24.824089 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820792 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:39:24.824089 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820795 4305 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:39:24.824089 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820799 4305 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:39:24.824089 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820801 4305 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:39:24.824089 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820804 4305 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:39:24.824089 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820813 4305 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:39:24.824089 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820816 4305 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:39:24.824599 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820818 4305 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:39:24.824599 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820821 4305 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:39:24.824599 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820823 4305 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:39:24.824599 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820826 4305 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:39:24.824599 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820829 4305 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:39:24.824599 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820831 4305 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:39:24.824599 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820834 4305 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:39:24.824599 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820837 4305 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:39:24.824599 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820840 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:39:24.824599 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820842 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:39:24.824599 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820845 4305 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:39:24.824599 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820847 4305 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:39:24.824599 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820850 4305 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:39:24.824599 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820852 4305 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:39:24.824599 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.820855 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:39:24.824599 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821366 4305 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:39:24.824599 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821371 4305 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:39:24.824599 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821374 4305 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:39:24.824599 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821377 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:39:24.825053 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821380 4305 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:39:24.825053 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821385 4305 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:39:24.825053 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821390 4305 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:39:24.825053 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821394 4305 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:39:24.825053 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821397 4305 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:39:24.825053 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821399 4305 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:39:24.825053 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821402 4305 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:39:24.825053 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821404 4305 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:39:24.825053 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821407 4305 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:39:24.825053 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821410 4305 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:39:24.825053 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821413 4305 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:39:24.825053 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821416 4305 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:39:24.825053 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821419 4305 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:39:24.825053 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821421 4305 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:39:24.825053 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821424 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:39:24.825053 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821427 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:39:24.825053 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821444 4305 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:39:24.825053 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821447 4305 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:39:24.825053 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821450 4305 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:39:24.825653 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821452 4305 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:39:24.825653 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821455 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:39:24.825653 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821457 4305 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:39:24.825653 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821460 4305 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:39:24.825653 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821462 4305 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:39:24.825653 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821465 4305 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:39:24.825653 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821468 4305 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:39:24.825653 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821470 4305 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:39:24.825653 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821473 4305 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:39:24.825653 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821475 4305 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:39:24.825653 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821478 4305 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:39:24.825653 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821481 4305 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:39:24.825653 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821483 4305 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:39:24.825653 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821486 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:39:24.825653 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821488 4305 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:39:24.825653 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821492 4305 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:39:24.825653 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821494 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:39:24.825653 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821497 4305 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:39:24.825653 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821499 4305 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:39:24.825653 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821502 4305 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:39:24.826535 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821504 4305 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:39:24.826535 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821507 4305 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:39:24.826535 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821510 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:39:24.826535 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821515 4305 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:39:24.826535 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821517 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:39:24.826535 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821520 4305 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:39:24.826535 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821522 4305 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:39:24.826535 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821525 4305 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:39:24.826535 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821527 4305 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:39:24.826535 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821530 4305 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:39:24.826535 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821532 4305 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:39:24.826535 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821535 4305 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:39:24.826535 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821537 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:39:24.826535 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821540 4305 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:39:24.826535 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821542 4305 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:39:24.826535 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821545 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:39:24.826535 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821547 4305 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:39:24.826535 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821549 4305 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:39:24.826535 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821552 4305 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:39:24.826535 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821554 4305 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:39:24.827409 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821557 4305 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:39:24.827409 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821560 4305 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:39:24.827409 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821563 4305 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:39:24.827409 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821565 4305 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:39:24.827409 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821568 4305 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:39:24.827409 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821570 4305 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:39:24.827409 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821573 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:39:24.827409 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821576 4305 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:39:24.827409 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821579 4305 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:39:24.827409 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821581 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:39:24.827409 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821584 4305 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:39:24.827409 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821586 4305 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:39:24.827409 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821588 4305 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:39:24.827409 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821591 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:39:24.827409 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821593 4305 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:39:24.827409 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821597 4305 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:39:24.827409 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821600 4305 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:39:24.827409 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821602 4305 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:39:24.827409 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821605 4305 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:39:24.828360 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821607 4305 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:39:24.828360 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821609 4305 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:39:24.828360 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821611 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:39:24.828360 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821614 4305 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:39:24.828360 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821616 4305 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:39:24.828360 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821619 4305 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:39:24.828360 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821621 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:39:24.828360 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821623 4305 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:39:24.828360 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821627 4305 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:39:24.828360 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821629 4305 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:39:24.828360 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821632 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:39:24.828360 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821635 4305 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:39:24.828360 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821637 4305 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:39:24.828360 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821640 4305 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:39:24.828360 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821642 4305 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:39:24.828360 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821644 4305 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:39:24.828360 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821649 4305 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:39:24.828360 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821653 4305 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:39:24.828360 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821655 4305 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:39:24.829111 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821658 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:39:24.829111 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821661 4305 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:39:24.829111 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821664 4305 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:39:24.829111 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821666 4305 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:39:24.829111 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.821669 4305 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:39:24.829111 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821767 4305 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 20:39:24.829111 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821774 4305 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 20:39:24.829111 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821780 4305 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 20:39:24.829111 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821786 4305 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 20:39:24.829111 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821790 4305 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 20:39:24.829111 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821794 4305 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 20:39:24.829111 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821798 4305 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 20:39:24.829111 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821802 4305 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 20:39:24.829111 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821805 4305 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 20:39:24.829111 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821809 4305 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 20:39:24.829111 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821812 4305 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 20:39:24.829111 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821815 4305 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 20:39:24.829111 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821819 4305 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 20:39:24.829111 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821822 4305 flags.go:64] FLAG: --cgroup-root="" Nov 03 20:39:24.829111 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821824 4305 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 20:39:24.829111 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821827 4305 flags.go:64] FLAG: --client-ca-file="" Nov 03 20:39:24.829111 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821830 4305 flags.go:64] FLAG: --cloud-provider="external" Nov 03 20:39:24.829111 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821834 4305 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 20:39:24.829111 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821838 4305 flags.go:64] FLAG: --cluster-domain="" Nov 03 20:39:24.829991 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821841 4305 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 20:39:24.829991 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821844 4305 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 20:39:24.829991 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821847 4305 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 20:39:24.829991 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821851 4305 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 20:39:24.829991 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821855 4305 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 20:39:24.829991 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821858 4305 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 20:39:24.829991 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821861 4305 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 20:39:24.829991 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821865 4305 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 20:39:24.829991 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821868 4305 flags.go:64] FLAG: --contention-profiling="false" Nov 03 20:39:24.829991 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821871 4305 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 20:39:24.829991 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821873 4305 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 20:39:24.829991 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821876 4305 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 20:39:24.829991 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821879 4305 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 20:39:24.829991 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821884 4305 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 20:39:24.829991 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821887 4305 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 20:39:24.829991 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821890 4305 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 20:39:24.829991 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821893 4305 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 20:39:24.829991 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821896 4305 flags.go:64] FLAG: --enable-server="true" Nov 03 20:39:24.829991 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821899 4305 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 20:39:24.829991 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821904 4305 flags.go:64] FLAG: --event-burst="100" Nov 03 20:39:24.829991 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821907 4305 flags.go:64] FLAG: --event-qps="50" Nov 03 20:39:24.829991 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821910 4305 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 20:39:24.829991 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821913 4305 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 20:39:24.829991 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821916 4305 flags.go:64] FLAG: --eviction-hard="" Nov 03 20:39:24.830602 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821920 4305 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 20:39:24.830602 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821923 4305 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 20:39:24.830602 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821926 4305 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 20:39:24.830602 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821929 4305 flags.go:64] FLAG: --eviction-soft="" Nov 03 20:39:24.830602 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821932 4305 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 20:39:24.830602 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821935 4305 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 20:39:24.830602 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821938 4305 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 20:39:24.830602 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821941 4305 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 20:39:24.830602 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821944 4305 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 20:39:24.830602 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821947 4305 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 20:39:24.830602 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821950 4305 flags.go:64] FLAG: --feature-gates="" Nov 03 20:39:24.830602 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821954 4305 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 20:39:24.830602 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821957 4305 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 20:39:24.830602 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821960 4305 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 20:39:24.830602 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821964 4305 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 20:39:24.830602 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821967 4305 flags.go:64] FLAG: --healthz-port="10248" Nov 03 20:39:24.830602 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821970 4305 flags.go:64] FLAG: --help="false" Nov 03 20:39:24.830602 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821973 4305 flags.go:64] FLAG: --hostname-override="ip-10-0-128-175.us-west-1.compute.internal" Nov 03 20:39:24.830602 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821976 4305 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 20:39:24.830602 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821979 4305 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 20:39:24.830602 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821982 4305 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 20:39:24.830602 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821985 4305 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 20:39:24.830602 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821989 4305 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 20:39:24.831164 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821991 4305 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 20:39:24.831164 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821994 4305 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 20:39:24.831164 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.821997 4305 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 20:39:24.831164 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822000 4305 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 20:39:24.831164 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822003 4305 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 20:39:24.831164 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822006 4305 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 20:39:24.831164 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822009 4305 flags.go:64] FLAG: --kube-reserved="" Nov 03 20:39:24.831164 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822012 4305 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 20:39:24.831164 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822015 4305 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 20:39:24.831164 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822018 4305 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 20:39:24.831164 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822021 4305 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 20:39:24.831164 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822023 4305 flags.go:64] FLAG: --lock-file="" Nov 03 20:39:24.831164 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822026 4305 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 20:39:24.831164 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822029 4305 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 20:39:24.831164 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822032 4305 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 20:39:24.831164 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822039 4305 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 20:39:24.831164 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822042 4305 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 20:39:24.831164 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822045 4305 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 20:39:24.831164 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822048 4305 flags.go:64] FLAG: --logging-format="text" Nov 03 20:39:24.831164 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822051 4305 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 20:39:24.831164 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822055 4305 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 20:39:24.831164 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822058 4305 flags.go:64] FLAG: --manifest-url="" Nov 03 20:39:24.831164 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822061 4305 flags.go:64] FLAG: --manifest-url-header="" Nov 03 20:39:24.831164 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822066 4305 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 20:39:24.831164 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822074 4305 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 20:39:24.831894 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822078 4305 flags.go:64] FLAG: --max-pods="110" Nov 03 20:39:24.831894 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822081 4305 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 20:39:24.831894 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822084 4305 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 20:39:24.831894 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822087 4305 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 20:39:24.831894 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822090 4305 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 20:39:24.831894 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822093 4305 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 20:39:24.831894 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822096 4305 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 20:39:24.831894 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822099 4305 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 20:39:24.831894 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822106 4305 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 20:39:24.831894 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822109 4305 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 20:39:24.831894 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822112 4305 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 20:39:24.831894 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822115 4305 flags.go:64] FLAG: --pod-cidr="" Nov 03 20:39:24.831894 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822118 4305 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 20:39:24.831894 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822122 4305 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 20:39:24.831894 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822125 4305 flags.go:64] FLAG: --pods-per-core="0" Nov 03 20:39:24.831894 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822128 4305 flags.go:64] FLAG: --port="10250" Nov 03 20:39:24.831894 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822131 4305 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 20:39:24.831894 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822134 4305 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-01cd9ab20b59b09db" Nov 03 20:39:24.831894 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822137 4305 flags.go:64] FLAG: --qos-reserved="" Nov 03 20:39:24.831894 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822140 4305 flags.go:64] FLAG: --read-only-port="10255" Nov 03 20:39:24.831894 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822143 4305 flags.go:64] FLAG: --register-node="true" Nov 03 20:39:24.831894 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822146 4305 flags.go:64] FLAG: --register-with-taints="" Nov 03 20:39:24.831894 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822151 4305 flags.go:64] FLAG: --registry-burst="10" Nov 03 20:39:24.831894 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822154 4305 flags.go:64] FLAG: --registry-qps="5" Nov 03 20:39:24.831894 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822157 4305 flags.go:64] FLAG: --reserved-cpus="" Nov 03 20:39:24.831894 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822160 4305 flags.go:64] FLAG: --reserved-memory="" Nov 03 20:39:24.832530 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822164 4305 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 20:39:24.832530 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822167 4305 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 20:39:24.832530 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822170 4305 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 20:39:24.832530 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822173 4305 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 20:39:24.832530 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822176 4305 flags.go:64] FLAG: --runonce="false" Nov 03 20:39:24.832530 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822180 4305 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 20:39:24.832530 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822183 4305 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 20:39:24.832530 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822186 4305 flags.go:64] FLAG: --seccomp-default="false" Nov 03 20:39:24.832530 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822189 4305 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 20:39:24.832530 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822191 4305 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 20:39:24.832530 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822195 4305 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 20:39:24.832530 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822197 4305 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 20:39:24.832530 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822201 4305 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 20:39:24.832530 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822204 4305 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 20:39:24.832530 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822206 4305 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 20:39:24.832530 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822209 4305 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 20:39:24.832530 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822212 4305 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 20:39:24.832530 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822215 4305 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 20:39:24.832530 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822218 4305 flags.go:64] FLAG: --system-cgroups="" Nov 03 20:39:24.832530 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822221 4305 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 20:39:24.832530 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822227 4305 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 20:39:24.832530 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822230 4305 flags.go:64] FLAG: --tls-cert-file="" Nov 03 20:39:24.832530 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822233 4305 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 20:39:24.832530 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822237 4305 flags.go:64] FLAG: --tls-min-version="" Nov 03 20:39:24.832530 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822240 4305 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 20:39:24.833139 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822243 4305 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 20:39:24.833139 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822246 4305 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 20:39:24.833139 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822249 4305 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 20:39:24.833139 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822253 4305 flags.go:64] FLAG: --v="2" Nov 03 20:39:24.833139 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822257 4305 flags.go:64] FLAG: --version="false" Nov 03 20:39:24.833139 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822262 4305 flags.go:64] FLAG: --vmodule="" Nov 03 20:39:24.833139 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822266 4305 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 20:39:24.833139 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822269 4305 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 20:39:24.833139 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822411 4305 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:39:24.833139 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822415 4305 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:39:24.833139 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822418 4305 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:39:24.833139 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822420 4305 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:39:24.833139 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822424 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:39:24.833139 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822427 4305 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:39:24.833139 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822443 4305 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:39:24.833139 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822446 4305 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:39:24.833139 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822449 4305 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:39:24.833139 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822451 4305 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:39:24.833139 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822454 4305 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:39:24.833139 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822456 4305 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:39:24.833139 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822459 4305 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:39:24.833717 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822461 4305 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:39:24.833717 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822464 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:39:24.833717 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822466 4305 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:39:24.833717 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822469 4305 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:39:24.833717 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822471 4305 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:39:24.833717 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822474 4305 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:39:24.833717 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822476 4305 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:39:24.833717 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822480 4305 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:39:24.833717 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822482 4305 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:39:24.833717 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822485 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:39:24.833717 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822488 4305 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:39:24.833717 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822490 4305 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:39:24.833717 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822492 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:39:24.833717 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822495 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:39:24.833717 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822499 4305 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:39:24.833717 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822502 4305 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:39:24.833717 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822504 4305 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:39:24.833717 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822507 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:39:24.833717 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822510 4305 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:39:24.833717 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822512 4305 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:39:24.834211 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822515 4305 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:39:24.834211 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822517 4305 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:39:24.834211 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822519 4305 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:39:24.834211 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822523 4305 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:39:24.834211 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822526 4305 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:39:24.834211 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822528 4305 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:39:24.834211 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822531 4305 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:39:24.834211 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822533 4305 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:39:24.834211 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822537 4305 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:39:24.834211 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822541 4305 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:39:24.834211 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822544 4305 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:39:24.834211 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822547 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:39:24.834211 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822550 4305 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:39:24.834211 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822552 4305 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:39:24.834211 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822555 4305 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:39:24.834211 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822557 4305 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:39:24.834211 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822560 4305 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:39:24.834211 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822563 4305 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:39:24.834211 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822565 4305 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:39:24.834211 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822568 4305 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:39:24.834908 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822571 4305 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:39:24.834908 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822574 4305 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:39:24.834908 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822577 4305 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:39:24.834908 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822579 4305 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:39:24.834908 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822581 4305 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:39:24.834908 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822584 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:39:24.834908 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822587 4305 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:39:24.834908 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822590 4305 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:39:24.834908 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822593 4305 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:39:24.834908 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822595 4305 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:39:24.834908 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822597 4305 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:39:24.834908 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822600 4305 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:39:24.834908 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822602 4305 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:39:24.834908 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822605 4305 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:39:24.834908 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822608 4305 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:39:24.834908 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822612 4305 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:39:24.834908 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822616 4305 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:39:24.834908 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822620 4305 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:39:24.834908 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822622 4305 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:39:24.835575 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822625 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:39:24.835575 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822628 4305 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:39:24.835575 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822630 4305 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:39:24.835575 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822633 4305 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:39:24.835575 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822636 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:39:24.835575 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822638 4305 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:39:24.835575 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822641 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:39:24.835575 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822643 4305 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:39:24.835575 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822646 4305 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:39:24.835575 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822648 4305 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:39:24.835575 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822650 4305 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:39:24.835575 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822653 4305 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:39:24.835575 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822655 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:39:24.835575 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822658 4305 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:39:24.835575 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822661 4305 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:39:24.835575 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822664 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:39:24.835575 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822666 4305 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:39:24.835575 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822669 4305 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:39:24.835575 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822671 4305 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:39:24.836133 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822675 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:39:24.836133 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822678 4305 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:39:24.836133 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822680 4305 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:39:24.836133 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822682 4305 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:39:24.836133 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822685 4305 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:39:24.836133 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822687 4305 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:39:24.836133 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822690 4305 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:39:24.836133 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822692 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:39:24.836133 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822695 4305 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:39:24.836133 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822699 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:39:24.836133 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822701 4305 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:39:24.836133 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822703 4305 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:39:24.836133 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822706 4305 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:39:24.836133 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822709 4305 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:39:24.836133 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822711 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:39:24.836133 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.822737 4305 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 20:39:24.836590 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822774 4305 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:39:24.836590 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822778 4305 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:39:24.836590 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822781 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:39:24.836590 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822784 4305 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:39:24.836590 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822786 4305 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:39:24.836590 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822789 4305 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:39:24.836590 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822791 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:39:24.836590 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822794 4305 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:39:24.836590 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822796 4305 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:39:24.836590 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822799 4305 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:39:24.836590 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822802 4305 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:39:24.836590 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822804 4305 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:39:24.836590 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822807 4305 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:39:24.836590 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822810 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:39:24.836590 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822813 4305 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:39:24.836590 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822815 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:39:24.836590 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822818 4305 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:39:24.836590 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822822 4305 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:39:24.836590 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822824 4305 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:39:24.836590 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822827 4305 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:39:24.837385 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822829 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:39:24.837385 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822832 4305 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:39:24.837385 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822834 4305 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:39:24.837385 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822837 4305 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:39:24.837385 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822839 4305 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:39:24.837385 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822843 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:39:24.837385 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822845 4305 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:39:24.837385 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822848 4305 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:39:24.837385 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822850 4305 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:39:24.837385 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822853 4305 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:39:24.837385 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822855 4305 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:39:24.837385 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822858 4305 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:39:24.837385 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822861 4305 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:39:24.837385 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822863 4305 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:39:24.837385 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822866 4305 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:39:24.837385 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822868 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:39:24.837385 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822871 4305 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:39:24.837385 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822873 4305 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:39:24.837385 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822875 4305 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:39:24.837385 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822878 4305 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:39:24.838417 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822880 4305 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:39:24.838417 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822883 4305 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:39:24.838417 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822885 4305 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:39:24.838417 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822888 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:39:24.838417 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822890 4305 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:39:24.838417 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822893 4305 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:39:24.838417 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822896 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:39:24.838417 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822898 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:39:24.838417 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822900 4305 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:39:24.838417 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822904 4305 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:39:24.838417 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822906 4305 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:39:24.838417 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822909 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:39:24.838417 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822911 4305 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:39:24.838417 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822914 4305 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:39:24.838417 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822917 4305 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:39:24.838417 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822919 4305 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:39:24.838417 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822922 4305 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:39:24.838417 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822925 4305 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:39:24.838417 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822928 4305 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:39:24.838417 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822930 4305 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:39:24.839060 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822933 4305 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:39:24.839060 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822935 4305 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:39:24.839060 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822939 4305 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:39:24.839060 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822942 4305 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:39:24.839060 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822945 4305 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:39:24.839060 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822948 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:39:24.839060 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822950 4305 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:39:24.839060 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822953 4305 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:39:24.839060 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822956 4305 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:39:24.839060 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822958 4305 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:39:24.839060 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822961 4305 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:39:24.839060 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822963 4305 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:39:24.839060 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822965 4305 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:39:24.839060 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822968 4305 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:39:24.839060 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822971 4305 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:39:24.839060 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822973 4305 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:39:24.839060 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822975 4305 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:39:24.839060 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822978 4305 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:39:24.839060 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822981 4305 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:39:24.839060 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822983 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:39:24.839637 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822986 4305 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:39:24.839637 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822989 4305 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:39:24.839637 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822992 4305 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:39:24.839637 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822994 4305 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:39:24.839637 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822997 4305 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:39:24.839637 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.822999 4305 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:39:24.839637 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.823002 4305 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:39:24.839637 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.823005 4305 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:39:24.839637 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.823007 4305 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:39:24.839637 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.823011 4305 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:39:24.839637 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.823015 4305 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:39:24.839637 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.823018 4305 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:39:24.839637 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.823021 4305 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:39:24.839637 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.823024 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:39:24.839637 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.823026 4305 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:39:24.839637 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.823029 4305 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:39:24.839637 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.823031 4305 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:39:24.839637 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.823034 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:39:24.839637 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.823036 4305 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:39:24.840134 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.823039 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:39:24.840134 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.823041 4305 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:39:24.840134 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.823044 4305 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:39:24.840134 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.823046 4305 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:39:24.840134 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.823049 4305 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:39:24.840134 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.823051 4305 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:39:24.840134 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.823054 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:39:24.840134 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.829641 4305 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 20:39:24.840134 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.829667 4305 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 20:39:24.840134 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.829694 4305 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 20:39:24.840134 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.829702 4305 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 20:39:24.840134 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.829985 4305 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:39:24.840134 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830104 4305 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:39:24.840134 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830112 4305 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:39:24.840134 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830117 4305 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:39:24.840134 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830121 4305 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:39:24.840134 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830124 4305 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:39:24.840134 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830127 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:39:24.840134 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830129 4305 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:39:24.840134 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830132 4305 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:39:24.840649 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830137 4305 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:39:24.840649 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830142 4305 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:39:24.840649 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830146 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:39:24.840649 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830149 4305 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:39:24.840649 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830152 4305 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:39:24.840649 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830155 4305 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:39:24.840649 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830158 4305 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:39:24.840649 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830161 4305 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:39:24.840649 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830163 4305 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:39:24.840649 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830166 4305 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:39:24.840649 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830168 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:39:24.840649 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830171 4305 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:39:24.840649 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830174 4305 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:39:24.840649 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830177 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:39:24.840649 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830180 4305 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:39:24.840649 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830182 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:39:24.840649 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830184 4305 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:39:24.840649 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830187 4305 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:39:24.840649 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830190 4305 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:39:24.841111 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830193 4305 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:39:24.841111 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830195 4305 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:39:24.841111 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830199 4305 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:39:24.841111 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830202 4305 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:39:24.841111 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830205 4305 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:39:24.841111 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830208 4305 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:39:24.841111 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830211 4305 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:39:24.841111 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830213 4305 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:39:24.841111 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830216 4305 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:39:24.841111 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830219 4305 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:39:24.841111 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830221 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:39:24.841111 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830224 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:39:24.841111 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830226 4305 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:39:24.841111 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830229 4305 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:39:24.841111 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830232 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:39:24.841111 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830234 4305 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:39:24.841111 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830237 4305 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:39:24.841111 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830240 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:39:24.841111 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830242 4305 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:39:24.841615 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830245 4305 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:39:24.841615 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830248 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:39:24.841615 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830251 4305 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:39:24.841615 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830253 4305 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:39:24.841615 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830256 4305 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:39:24.841615 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830259 4305 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:39:24.841615 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830262 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:39:24.841615 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830264 4305 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:39:24.841615 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830267 4305 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:39:24.841615 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830269 4305 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:39:24.841615 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830272 4305 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:39:24.841615 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830274 4305 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:39:24.841615 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830277 4305 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:39:24.841615 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830280 4305 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:39:24.841615 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830282 4305 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:39:24.841615 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830285 4305 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:39:24.841615 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830289 4305 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:39:24.841615 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830292 4305 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:39:24.841615 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830294 4305 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:39:24.841615 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830299 4305 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:39:24.842099 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830302 4305 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:39:24.842099 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830304 4305 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:39:24.842099 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830307 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:39:24.842099 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830309 4305 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:39:24.842099 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830312 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:39:24.842099 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830314 4305 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:39:24.842099 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830317 4305 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:39:24.842099 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830319 4305 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:39:24.842099 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830322 4305 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:39:24.842099 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830324 4305 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:39:24.842099 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830328 4305 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:39:24.842099 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830330 4305 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:39:24.842099 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830333 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:39:24.842099 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830336 4305 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:39:24.842099 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830339 4305 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:39:24.842099 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830342 4305 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:39:24.842099 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830344 4305 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:39:24.842099 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830347 4305 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:39:24.842099 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830350 4305 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:39:24.842099 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830352 4305 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:39:24.842607 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830356 4305 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:39:24.842607 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830359 4305 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:39:24.842607 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830362 4305 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:39:24.842607 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830365 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:39:24.842607 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830368 4305 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:39:24.842607 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830371 4305 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:39:24.842607 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830373 4305 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:39:24.842607 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830376 4305 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:39:24.842607 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830379 4305 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:39:24.842607 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830381 4305 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:39:24.842607 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830384 4305 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:39:24.842607 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830386 4305 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:39:24.842607 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830389 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:39:24.842607 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830392 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:39:24.842607 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830394 4305 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:39:24.842607 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830397 4305 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:39:24.842607 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830399 4305 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:39:24.842607 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830402 4305 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:39:24.842607 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830405 4305 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:39:24.843069 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.830458 4305 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 20:39:24.843069 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830630 4305 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:39:24.843069 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830636 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:39:24.843069 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830640 4305 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:39:24.843069 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830642 4305 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:39:24.843069 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830645 4305 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:39:24.843069 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830655 4305 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:39:24.843069 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830658 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:39:24.843069 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830661 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:39:24.843069 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830663 4305 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:39:24.843069 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830666 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:39:24.843069 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830669 4305 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:39:24.843069 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830671 4305 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:39:24.843069 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830674 4305 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:39:24.843069 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830676 4305 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:39:24.843069 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830679 4305 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:39:24.843490 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830681 4305 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:39:24.843490 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830684 4305 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:39:24.843490 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830687 4305 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:39:24.843490 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830690 4305 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:39:24.843490 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830692 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:39:24.843490 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830695 4305 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:39:24.843490 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830697 4305 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:39:24.843490 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830700 4305 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:39:24.843490 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830703 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:39:24.843490 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830705 4305 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:39:24.843490 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830709 4305 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:39:24.843490 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830713 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:39:24.843490 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830717 4305 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:39:24.843490 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830720 4305 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:39:24.843490 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830723 4305 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:39:24.843490 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830726 4305 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:39:24.843490 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830729 4305 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:39:24.843490 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830731 4305 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:39:24.843490 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830734 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:39:24.843954 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830737 4305 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:39:24.843954 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830739 4305 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:39:24.843954 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830742 4305 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:39:24.843954 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830744 4305 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:39:24.843954 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830752 4305 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:39:24.843954 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830755 4305 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:39:24.843954 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830758 4305 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:39:24.843954 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830760 4305 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:39:24.843954 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830763 4305 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:39:24.843954 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830767 4305 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:39:24.843954 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830770 4305 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:39:24.843954 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830773 4305 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:39:24.843954 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830776 4305 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:39:24.843954 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830779 4305 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:39:24.843954 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830781 4305 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:39:24.843954 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830784 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:39:24.843954 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830787 4305 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:39:24.843954 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830790 4305 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:39:24.843954 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830792 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:39:24.844410 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830795 4305 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:39:24.844410 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830797 4305 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:39:24.844410 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830800 4305 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:39:24.844410 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830802 4305 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:39:24.844410 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830805 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:39:24.844410 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830807 4305 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:39:24.844410 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830810 4305 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:39:24.844410 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830812 4305 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:39:24.844410 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830815 4305 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:39:24.844410 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830817 4305 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:39:24.844410 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830820 4305 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:39:24.844410 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830823 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:39:24.844410 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830825 4305 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:39:24.844410 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830828 4305 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:39:24.844410 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830830 4305 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:39:24.844410 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830833 4305 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:39:24.844410 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830835 4305 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:39:24.844410 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830837 4305 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:39:24.844410 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830841 4305 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:39:24.844894 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830843 4305 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:39:24.844894 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830846 4305 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:39:24.844894 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830848 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:39:24.844894 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830851 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:39:24.844894 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830853 4305 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:39:24.844894 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830856 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:39:24.844894 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830858 4305 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:39:24.844894 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830861 4305 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:39:24.844894 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830863 4305 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:39:24.844894 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830866 4305 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:39:24.844894 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830869 4305 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:39:24.844894 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830871 4305 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:39:24.844894 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830874 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:39:24.844894 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830876 4305 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:39:24.844894 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830879 4305 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:39:24.844894 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830881 4305 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:39:24.844894 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830884 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:39:24.844894 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830886 4305 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:39:24.844894 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830889 4305 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:39:24.845369 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830891 4305 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:39:24.845369 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830894 4305 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:39:24.845369 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830896 4305 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:39:24.845369 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830899 4305 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:39:24.845369 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830902 4305 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:39:24.845369 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830904 4305 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:39:24.845369 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830907 4305 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:39:24.845369 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830909 4305 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:39:24.845369 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830911 4305 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:39:24.845369 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830914 4305 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:39:24.845369 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830917 4305 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:39:24.845369 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830919 4305 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:39:24.845369 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830922 4305 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:39:24.845369 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830925 4305 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:39:24.845369 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830927 4305 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:39:24.845369 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.830951 4305 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 20:39:24.845768 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830990 4305 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:39:24.845768 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830995 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:39:24.845768 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.830999 4305 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:39:24.845768 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831003 4305 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:39:24.845768 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831006 4305 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:39:24.845768 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831009 4305 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:39:24.845768 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831011 4305 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:39:24.845768 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831014 4305 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:39:24.845768 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831016 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:39:24.845768 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831019 4305 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:39:24.845768 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831021 4305 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:39:24.845768 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831024 4305 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:39:24.845768 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831026 4305 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:39:24.845768 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831028 4305 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:39:24.845768 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831031 4305 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:39:24.845768 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831034 4305 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:39:24.845768 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831036 4305 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:39:24.845768 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831039 4305 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:39:24.845768 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831043 4305 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:39:24.846216 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831047 4305 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:39:24.846216 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831051 4305 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:39:24.846216 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831053 4305 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:39:24.846216 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831056 4305 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:39:24.846216 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831058 4305 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:39:24.846216 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831061 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:39:24.846216 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831063 4305 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:39:24.846216 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831065 4305 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:39:24.846216 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831068 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:39:24.846216 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831070 4305 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:39:24.846216 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831072 4305 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:39:24.846216 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831075 4305 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:39:24.846216 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831078 4305 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:39:24.846216 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831081 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:39:24.846216 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831084 4305 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:39:24.846216 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831089 4305 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:39:24.846216 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831092 4305 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:39:24.846216 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831094 4305 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:39:24.846216 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831097 4305 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:39:24.846743 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831100 4305 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:39:24.846743 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831102 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:39:24.846743 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831104 4305 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:39:24.846743 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831107 4305 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:39:24.846743 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831109 4305 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:39:24.846743 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831112 4305 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:39:24.846743 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831115 4305 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:39:24.846743 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831117 4305 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:39:24.846743 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831119 4305 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:39:24.846743 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831122 4305 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:39:24.846743 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831125 4305 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:39:24.846743 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831128 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:39:24.846743 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831130 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:39:24.846743 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831133 4305 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:39:24.846743 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831135 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:39:24.846743 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831138 4305 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:39:24.846743 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831141 4305 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:39:24.846743 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831143 4305 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:39:24.846743 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831146 4305 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:39:24.846743 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831148 4305 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:39:24.847468 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831151 4305 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:39:24.847468 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831153 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:39:24.847468 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831156 4305 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:39:24.847468 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831158 4305 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:39:24.847468 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831160 4305 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:39:24.847468 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831164 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:39:24.847468 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831166 4305 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:39:24.847468 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831169 4305 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:39:24.847468 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831173 4305 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:39:24.847468 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831176 4305 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:39:24.847468 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831179 4305 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:39:24.847468 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831182 4305 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:39:24.847468 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831184 4305 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:39:24.847468 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831187 4305 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:39:24.847468 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831190 4305 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:39:24.847468 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831193 4305 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:39:24.847468 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831195 4305 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:39:24.847468 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831198 4305 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:39:24.847468 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831200 4305 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:39:24.847468 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831203 4305 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:39:24.848231 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831206 4305 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:39:24.848231 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831208 4305 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:39:24.848231 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831210 4305 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:39:24.848231 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831214 4305 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:39:24.848231 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831216 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:39:24.848231 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831219 4305 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:39:24.848231 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831221 4305 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:39:24.848231 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831224 4305 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:39:24.848231 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831226 4305 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:39:24.848231 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831229 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:39:24.848231 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831232 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:39:24.848231 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831234 4305 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:39:24.848231 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831236 4305 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:39:24.848231 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831239 4305 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:39:24.848231 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831241 4305 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:39:24.848231 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831244 4305 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:39:24.848231 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831246 4305 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:39:24.848231 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831249 4305 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:39:24.848231 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831251 4305 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:39:24.848231 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831254 4305 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:39:24.849068 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831258 4305 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:39:24.849068 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831261 4305 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:39:24.849068 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831263 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:39:24.849068 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831266 4305 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:39:24.849068 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831269 4305 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:39:24.849068 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831272 4305 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:39:24.849068 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831275 4305 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:39:24.849068 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:24.831278 4305 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:39:24.849068 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.831345 4305 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 20:39:24.849068 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.832471 4305 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 03 20:39:24.849068 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.832550 4305 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 03 20:39:24.849068 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.832736 4305 server.go:1014] "Starting client certificate rotation" Nov 03 20:39:24.849068 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.832874 4305 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 20:39:24.849068 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.832938 4305 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 20:33:46 +0000 UTC" deadline="2028-04-10 17:44:52.007012782 +0000 UTC" Nov 03 20:39:24.849068 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.832969 4305 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12573h5m27.174046789s" Nov 03 20:39:24.849068 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.833239 4305 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 20:39:24.849557 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.834631 4305 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 20:39:24.849557 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.836309 4305 log.go:25] "Validated CRI v1 runtime API" Nov 03 20:39:24.849557 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.838186 4305 log.go:25] "Validated CRI v1 image API" Nov 03 20:39:24.849557 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.838945 4305 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 20:39:24.849557 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.840366 4305 fs.go:135] Filesystem UUIDs: map[052e3f40-4b72-478f-bc3f-d4be5679e7e8:/dev/nvme0n1p3 69cfa50c-3477-4765-b594-c223f3a188f4:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Nov 03 20:39:24.849704 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.840382 4305 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0244c122743d370f3b669ba61dda5e3b66c9cbafc104f327f0f7319575e86e30/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0244c122743d370f3b669ba61dda5e3b66c9cbafc104f327f0f7319575e86e30/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0fb1e6fcef76d2b173be8c0c5306a824f9f80d463e0267b5c4299accdb3b189e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0fb1e6fcef76d2b173be8c0c5306a824f9f80d463e0267b5c4299accdb3b189e/userdata/shm major:0 minor:73 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1ff43da0fa130ddee036756b0058aaa333d6b988c333f92a80145ca74248dd06/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1ff43da0fa130ddee036756b0058aaa333d6b988c333f92a80145ca74248dd06/userdata/shm major:0 minor:201 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/335f16e7434a1f6f00cadb6dafbd0ed781f13d71b72ded6c24a31004d704cbf3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/335f16e7434a1f6f00cadb6dafbd0ed781f13d71b72ded6c24a31004d704cbf3/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6295278154ec9f50f8125c2ddc58308bc6b1e57f7a7db0f1e6a0d3cc5a26b298/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6295278154ec9f50f8125c2ddc58308bc6b1e57f7a7db0f1e6a0d3cc5a26b298/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/81847b395d92df6e6a5c82d000e7362957027e3524f065444da36d85198c98db/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/81847b395d92df6e6a5c82d000e7362957027e3524f065444da36d85198c98db/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/90c4be718888538813c8970ee006a8aacbc94a7b254ed0aa3bc6b4bdcb72b2be/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/90c4be718888538813c8970ee006a8aacbc94a7b254ed0aa3bc6b4bdcb72b2be/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ad5cd83d0b08d50512686eab4cbec286050d49546867978894e38f67ed35fc74/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ad5cd83d0b08d50512686eab4cbec286050d49546867978894e38f67ed35fc74/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c43e801572ab4ba97c6f192f69c89cab85583a68c7ae5d61899f09c02737f8b4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c43e801572ab4ba97c6f192f69c89cab85583a68c7ae5d61899f09c02737f8b4/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d7d2df7f30348457242061c096e76a42dad7a50762bf9b4945e6f2e08467c118/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d7d2df7f30348457242061c096e76a42dad7a50762bf9b4945e6f2e08467c118/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f8d26f4dc3ef29cbb33bb1657db2beddcc5187dd78248b6593e7c3f469d4499e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f8d26f4dc3ef29cbb33bb1657db2beddcc5187dd78248b6593e7c3f469d4499e/userdata/shm major:0 minor:89 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fc504e8ae03bf81d454277ab0a6d074765ef9f32dcb03c2bb8d8b38f2e759ba3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fc504e8ae03bf81d454277ab0a6d074765ef9f32dcb03c2bb8d8b38f2e759ba3/userdata/shm major:0 minor:77 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/03985fe9-0dfe-4f9a-97db-bb4c0298be5c/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/03985fe9-0dfe-4f9a-97db-bb4c0298be5c/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12658495-b5af-4c01-9c72-19b1173f7529/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/12658495-b5af-4c01-9c72-19b1173f7529/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12658495-b5af-4c01-9c72-19b1173f7529/volumes/kubernetes.io~projected/kube-api-access-6cscp:{mountpoint:/var/lib/kubelet/pods/12658495-b5af-4c01-9c72-19b1173f7529/volumes/kubernetes.io~projected/kube-api-access-6cscp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12658495-b5af-4c01-9c72-19b1173f7529/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/12658495-b5af-4c01-9c72-19b1173f7529/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2dfb2512-b394-4a37-a5ba-a783d4e46c46/volumes/kubernetes.io~projected/kube-api-access-5rlc8:{mountpoint:/var/lib/kubelet/pods/2dfb2512-b394-4a37-a5ba-a783d4e46c46/volumes/kubernetes.io~projected/kube-api-access-5rlc8 major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2f1f5ba6-ccc5-4f0d-a135-743747ce667a/volumes/kubernetes.io~projected/kube-api-access-z294f:{mountpoint:/var/lib/kubelet/pods/2f1f5ba6-ccc5-4f0d-a135-743747ce667a/volumes/kubernetes.io~projected/kube-api-access-z294f major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3e72f317-1042-40b0-b709-dc3d650754b5/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/3e72f317-1042-40b0-b709-dc3d650754b5/volumes/kubernetes.io~secret/agent-certs major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4a5df9f4-aaaa-4441-a2e0-28744e197828/volumes/kubernetes.io~projected/kube-api-access-z2vcp:{mountpoint:/var/lib/kubelet/pods/4a5df9f4-aaaa-4441-a2e0-28744e197828/volumes/kubernetes.io~projected/kube-api-access-z2vcp major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6d81822a-bda4-4f5e-a5f4-05e1606f22bf/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/6d81822a-bda4-4f5e-a5f4-05e1606f22bf/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6d81822a-bda4-4f5e-a5f4-05e1606f22bf/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/6d81822a-bda4-4f5e-a5f4-05e1606f22bf/volumes/kubernetes.io~empty-dir/tmp major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6d81822a-bda4-4f5e-a5f4-05e1606f22bf/volumes/kubernetes.io~projected/kube-api-access-wj57q:{mountpoint:/var/lib/kubelet/pods/6d81822a-bda4-4f5e-a5f4-05e1606f22bf/volumes/kubernetes.io~projected/kube-api-access-wj57q major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6e89ebac-2191-4acc-b4d4-ff326b3b74ac/volumes/kubernetes.io~projected/kube-api-access-tmbq4:{mountpoint:/var/lib/kubelet/pods/6e89ebac-2191-4acc-b4d4-ff326b3b74ac/volumes/kubernetes.io~projected/kube-api-access-tmbq4 major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/747345eb-a407-423b-92df-c5d765a7da2d/volumes/kubernetes.io~projected/kube-api-access-l2brl:{mountpoint:/var/lib/kubelet/pods/747345eb-a407-423b-92df-c5d765a7da2d/volumes/kubernetes.io~projected/kube-api-access-l2brl major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7ced550a-b3cb-4605-b8dc-aab3b55804b4/volumes/kubernetes.io~projected/kube-api-access-t6rjq:{mountpoint:/var/lib/kubelet/pods/7ced550a-b3cb-4605-b8dc-aab3b55804b4/volumes/kubernetes.io~projected/kube-api-access-t6rjq major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bafd96d7-6982-4c93-81fc-d61643002fd1/volumes/kubernetes.io~projected/kube-api-access-25sxl:{mountpoint:/var/lib/kubelet/pods/bafd96d7-6982-4c93-81fc-d61643002fd1/volumes/kubernetes.io~projected/kube-api-access-25sxl major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cf15bf7c-07c6-4d82-987b-aa4a332d5ca9/volumes/kubernetes.io~projected/kube-api-access-t8b6m:{mountpoint:/var/lib/kubelet/pods/cf15bf7c-07c6-4d82-987b-aa4a332d5ca9/volumes/kubernetes.io~projected/kube-api-access-t8b6m major:0 minor:59 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/c4607a4d81a6018bc2759cb358bc42b69197fea714f99e4f4d949aae0563c273/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/881707bcde85421c6d5856d4aae5598b684248e9dfa23e366cf4a66d9656fb59/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/5309ed4da18704452e06dfb024c358eb53c6c6e3748ab89b32fcc635037e3207/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/066209bd15093235b7037be210517669acffb0282b78bb5034f3aa8ea553e7c9/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/6cbc8d7d92025cf3e5d5cb5b9a19c8b89e3229e008f2bd1d174cae6640409fc8/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/0174ee8b67834558643d3ed1473d767f8089048a94ca2adeaa7f90c3df4fe314/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/e758e7c04fb38890b6728aa143e019e6504b0c6715c23142b76cf1fe3ac08714/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/c6f3f25071d8d5b129bc67135976a07657f591245ae99424df4a43efcb787cb2/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/ca22e63bf97069ca7e9f2fbb21186b088fdce69d6f99508601166670fbabeae3/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-131:{mountpoint:/var/lib/containers/storage/overlay/e2a088808883fbb3d8417751244a9c48bad9b06bd4fdf8ac7d92435f9320d5c1/merged major:0 minor:131 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/bde2f92f20768dac3bb27ad641e3ecfa17a2d7ddf9a1e022403e1c6533133c48/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/4f17a15227eabd09befa7234d08923937a9e769de47cbde0ea973cddfc33337a/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/3b7c065639c6f9ae798d59b650f8a0b8b60de9cf4c2f40ae98a9ad7029db4049/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/0df6590881059a722de928467d6fca52b2935f4e4796e05831610f602b9f33eb/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-159:{mountpoint:/var/lib/containers/storage/overlay/1e909cfab0e4756401e08995538a20e5673336a4ec2ef479c394764095ee59ec/merged major:0 minor:159 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/23f52fd84d352febf7755300d6882629d334ecd88501989b314d46bdb6b30204/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/6e1c046f5e2ec7049f537404bab3e5dad1d6d0d47a36118e9bab419f20a30c41/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/d364251340170ec4dd195aa1e9b2f2b9bd2bd093216d46ee2dea90c75e72413e/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/3d91bc7d16f91df598b789a11b93c551aaf593d8d7781c218fc7ab668cf66b7b/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/2fd6b2f83f017c1fc7fb8f2ede63ee41ba66a9132d16c656c94b34c7908fab92/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/c8c162c6e3f2265b205857a26063983503f12b583b38c8367d73ae25ed954cf3/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-205:{mountpoint:/var/lib/containers/storage/overlay/fb5c27e8df7cbb512beeb489bc082e0fa5010e77d19ee649932b67c56b8b1d73/merged major:0 minor:205 fsType:overlay blockSize:0} overlay_0-207:{mountpoint:/var/lib/containers/storage/overlay/8760e5867e8213e9a3157203eef8335ca5f0d85486828afca9395c3063839ba5/merged major:0 minor:207 fsType:overlay blockSize:0} overlay_0-209:{mountpoint:/var/lib/containers/storage/overlay/206a0c3d8cbdde3367019be4fef1dd8fa1a495b0382f22e681f0a50528f86456/merged major:0 minor:209 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/bf1bd46cda9fce68752852549bcf9a588f8908c32a42d6c6c43aaf408dc97cac/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/0d99f91c59d318cb1aafbab1c867a2da873caf64637242614d2f30d23a320460/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/55cd1985a313bb5767926302d12330f4ab367ef7e9ab2bafe7a3427e59691c47/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/18505d8951afe2d59073187af0ecad730fa4ffc09bf7caa021f820b7e5bca8a4/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-75:{mountpoint:/var/lib/containers/storage/overlay/40e92bd074e3d211a9a5ff556a47b40eabadba42dfbee7020906a9f0c9728538/merged major:0 minor:75 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/ea75b61684f1232a931347e90bb3b1cefa9563c967b2be9f22ebcd08e76c4699/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/3ff920219e2224d88395824c4996a9f85f65d69e627548b9b3c845b8f84d2569/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/26fd20db5bbce08812b17684668a8404ea55e9d70f2678f5a7fb04f62be1c568/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-91:{mountpoint:/var/lib/containers/storage/overlay/ef41f3479d15a5771122f455804c2f492479b97a2becbe042349a82b183ee0dc/merged major:0 minor:91 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/db4b5973b3e853ceafce7d3349830f8027fe80a1a0bd348e420a2b26a889cf1b/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/c90b637a09365028d64e3990e14ab81f723da68d99a1250fa80cb863df163567/merged major:0 minor:98 fsType:overlay blockSize:0}] Nov 03 20:39:24.849704 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.846506 4305 manager.go:217] Machine: {Timestamp:2026-11-03 20:39:24.845966363 +0000 UTC m=+0.071286146 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098560 MemoryCapacity:32812007424 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e10b2ff5b3ea62e589b49f2e14a92 SystemUUID:ec2e10b2-ff5b-3ea6-2e58-9b49f2e14a92 BootID:6840dd11-ad1f-4708-8cfc-1d96c63aaed5 Filesystems:[{Device:/run/containers/storage/overlay-containers/81847b395d92df6e6a5c82d000e7362957027e3524f065444da36d85198c98db/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/6e89ebac-2191-4acc-b4d4-ff326b3b74ac/volumes/kubernetes.io~projected/kube-api-access-tmbq4 DeviceMajor:0 DeviceMinor:54 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406003712 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0244c122743d370f3b669ba61dda5e3b66c9cbafc104f327f0f7319575e86e30/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6d81822a-bda4-4f5e-a5f4-05e1606f22bf/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:48 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/90c4be718888538813c8970ee006a8aacbc94a7b254ed0aa3bc6b4bdcb72b2be/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2f1f5ba6-ccc5-4f0d-a135-743747ce667a/volumes/kubernetes.io~projected/kube-api-access-z294f DeviceMajor:0 DeviceMinor:51 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/747345eb-a407-423b-92df-c5d765a7da2d/volumes/kubernetes.io~projected/kube-api-access-l2brl DeviceMajor:0 DeviceMinor:57 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/12658495-b5af-4c01-9c72-19b1173f7529/volumes/kubernetes.io~projected/kube-api-access-6cscp DeviceMajor:0 DeviceMinor:55 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ad5cd83d0b08d50512686eab4cbec286050d49546867978894e38f67ed35fc74/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-209 DeviceMajor:0 DeviceMinor:209 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4a5df9f4-aaaa-4441-a2e0-28744e197828/volumes/kubernetes.io~projected/kube-api-access-z2vcp DeviceMajor:0 DeviceMinor:53 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/6d81822a-bda4-4f5e-a5f4-05e1606f22bf/volumes/kubernetes.io~projected/kube-api-access-wj57q DeviceMajor:0 DeviceMinor:56 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/cf15bf7c-07c6-4d82-987b-aa4a332d5ca9/volumes/kubernetes.io~projected/kube-api-access-t8b6m DeviceMajor:0 DeviceMinor:59 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6295278154ec9f50f8125c2ddc58308bc6b1e57f7a7db0f1e6a0d3cc5a26b298/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/12658495-b5af-4c01-9c72-19b1173f7529/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-207 DeviceMajor:0 DeviceMinor:207 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0fb1e6fcef76d2b173be8c0c5306a824f9f80d463e0267b5c4299accdb3b189e/userdata/shm DeviceMajor:0 DeviceMinor:73 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3e72f317-1042-40b0-b709-dc3d650754b5/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:49 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/bafd96d7-6982-4c93-81fc-d61643002fd1/volumes/kubernetes.io~projected/kube-api-access-25sxl DeviceMajor:0 DeviceMinor:52 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-131 DeviceMajor:0 DeviceMinor:131 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2dfb2512-b394-4a37-a5ba-a783d4e46c46/volumes/kubernetes.io~projected/kube-api-access-5rlc8 DeviceMajor:0 DeviceMinor:197 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f8d26f4dc3ef29cbb33bb1657db2beddcc5187dd78248b6593e7c3f469d4499e/userdata/shm DeviceMajor:0 DeviceMinor:89 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-91 DeviceMajor:0 DeviceMinor:91 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1ff43da0fa130ddee036756b0058aaa333d6b988c333f92a80145ca74248dd06/userdata/shm DeviceMajor:0 DeviceMinor:201 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-205 DeviceMajor:0 DeviceMinor:205 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/12658495-b5af-4c01-9c72-19b1173f7529/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:50 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/03985fe9-0dfe-4f9a-97db-bb4c0298be5c/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:45 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/6d81822a-bda4-4f5e-a5f4-05e1606f22bf/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:47 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/7ced550a-b3cb-4605-b8dc-aab3b55804b4/volumes/kubernetes.io~projected/kube-api-access-t6rjq DeviceMajor:0 DeviceMinor:58 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fc504e8ae03bf81d454277ab0a6d074765ef9f32dcb03c2bb8d8b38f2e759ba3/userdata/shm DeviceMajor:0 DeviceMinor:77 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/335f16e7434a1f6f00cadb6dafbd0ed781f13d71b72ded6c24a31004d704cbf3/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d7d2df7f30348457242061c096e76a42dad7a50762bf9b4945e6f2e08467c118/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-75 DeviceMajor:0 DeviceMinor:75 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-159 DeviceMajor:0 DeviceMinor:159 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c43e801572ab4ba97c6f192f69c89cab85583a68c7ae5d61899f09c02737f8b4/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1ff43da0fa130dd MacAddress:5a:76:a3:56:20:a0 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:15:22:95:81:11 Speed:0 Mtu:9001} {Name:br-int MacAddress:a6:48:c0:27:2f:06 Speed:0 Mtu:8901} {Name:c43e801572ab4ba MacAddress:4a:b4:fe:05:fb:1d Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:15:22:95:81:11 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:16:de:4c:7c:ff:fc Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:96:61:b7:b5:b6:c1 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812007424 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 20:39:24.849704 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.846727 4305 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 20:39:24.849704 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.846773 4305 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 20:39:24.849704 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.848038 4305 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 20:39:24.849704 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.848099 4305 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-175.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 20:39:24.849704 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.848649 4305 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 20:39:24.849704 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.848660 4305 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 20:39:24.849704 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.848672 4305 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 20:39:24.849704 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.848686 4305 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 20:39:24.849704 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.848790 4305 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 20:39:24.849704 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.848953 4305 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 20:39:24.849704 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.849063 4305 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 20:39:24.849704 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.849098 4305 kubelet.go:493] "Attempting to sync node with API server" Nov 03 20:39:24.849704 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.849109 4305 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 20:39:24.850376 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.849119 4305 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 20:39:24.850376 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.849128 4305 kubelet.go:405] "Adding apiserver pod source" Nov 03 20:39:24.850376 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.849136 4305 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 20:39:24.850376 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.849731 4305 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 20:39:24.850376 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.849740 4305 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 20:39:24.850376 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.850237 4305 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 20:39:24.852412 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.852399 4305 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 20:39:24.852412 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.852412 4305 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 20:39:24.852545 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.852535 4305 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 03 20:39:24.852754 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.852743 4305 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 20:39:24.852790 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.852773 4305 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 20:39:24.852876 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.852867 4305 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 20:39:24.852907 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.852882 4305 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 20:39:24.852907 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.852889 4305 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 20:39:24.852907 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.852898 4305 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 20:39:24.852907 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.852905 4305 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 20:39:24.853018 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.852912 4305 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 20:39:24.853018 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.852919 4305 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 20:39:24.853018 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.852926 4305 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 20:39:24.853018 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.852934 4305 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 20:39:24.853018 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.852942 4305 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 20:39:24.853018 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.852951 4305 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 20:39:24.853018 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.852961 4305 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 20:39:24.853018 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.852989 4305 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 20:39:24.853018 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.852996 4305 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 20:39:24.856603 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.856524 4305 server.go:1263] "Started kubelet" Nov 03 20:39:24.856713 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.856583 4305 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 20:39:24.856769 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.856698 4305 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 20:39:24.856821 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.856794 4305 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 20:39:24.857084 ip-10-0-128-175 systemd[1]: Started Kubernetes Kubelet. Nov 03 20:39:24.857278 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.857228 4305 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 20:39:24.857853 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.857838 4305 server.go:317] "Adding debug handlers to kubelet server" Nov 03 20:39:24.859307 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.859021 4305 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 20:39:24.859307 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.859240 4305 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 20:39:24.859307 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.859250 4305 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 20:39:24.861768 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.859295 4305 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 20:39:24.861768 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.860461 4305 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 20:39:24.861768 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.860541 4305 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 20:33:47 +0000 UTC" deadline="2028-06-29 10:02:16.573469446 +0000 UTC" Nov 03 20:39:24.861768 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:39:24.860543 4305 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-175.us-west-1.compute.internal\" not found" Nov 03 20:39:24.861768 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.860613 4305 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14485h22m51.71286057s" Nov 03 20:39:24.867083 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.866042 4305 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6d81822a-bda4-4f5e-a5f4-05e1606f22bf" volumeName="kubernetes.io/projected/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-kube-api-access-wj57q" seLinuxMountContext="" Nov 03 20:39:24.867083 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.866459 4305 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03985fe9-0dfe-4f9a-97db-bb4c0298be5c" volumeName="kubernetes.io/secret/03985fe9-0dfe-4f9a-97db-bb4c0298be5c-original-pull-secret" seLinuxMountContext="" Nov 03 20:39:24.867083 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.866483 4305 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12658495-b5af-4c01-9c72-19b1173f7529" volumeName="kubernetes.io/configmap/12658495-b5af-4c01-9c72-19b1173f7529-ovnkube-config" seLinuxMountContext="" Nov 03 20:39:24.867083 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.866496 4305 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12658495-b5af-4c01-9c72-19b1173f7529" volumeName="kubernetes.io/projected/12658495-b5af-4c01-9c72-19b1173f7529-kube-api-access-6cscp" seLinuxMountContext="" Nov 03 20:39:24.867083 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.866508 4305 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12658495-b5af-4c01-9c72-19b1173f7529" volumeName="kubernetes.io/secret/12658495-b5af-4c01-9c72-19b1173f7529-ovn-node-metrics-cert" seLinuxMountContext="" Nov 03 20:39:24.867083 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.866519 4305 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f1f5ba6-ccc5-4f0d-a135-743747ce667a" volumeName="kubernetes.io/configmap/2f1f5ba6-ccc5-4f0d-a135-743747ce667a-cni-sysctl-allowlist" seLinuxMountContext="" Nov 03 20:39:24.867083 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.866535 4305 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e72f317-1042-40b0-b709-dc3d650754b5" volumeName="kubernetes.io/secret/3e72f317-1042-40b0-b709-dc3d650754b5-agent-certs" seLinuxMountContext="" Nov 03 20:39:24.867083 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.866546 4305 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4a5df9f4-aaaa-4441-a2e0-28744e197828" volumeName="kubernetes.io/configmap/4a5df9f4-aaaa-4441-a2e0-28744e197828-cni-binary-copy" seLinuxMountContext="" Nov 03 20:39:24.867083 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.866578 4305 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e89ebac-2191-4acc-b4d4-ff326b3b74ac" volumeName="kubernetes.io/projected/6e89ebac-2191-4acc-b4d4-ff326b3b74ac-kube-api-access-tmbq4" seLinuxMountContext="" Nov 03 20:39:24.867083 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.866590 4305 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f1f5ba6-ccc5-4f0d-a135-743747ce667a" volumeName="kubernetes.io/configmap/2f1f5ba6-ccc5-4f0d-a135-743747ce667a-cni-binary-copy" seLinuxMountContext="" Nov 03 20:39:24.867083 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.866604 4305 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f1f5ba6-ccc5-4f0d-a135-743747ce667a" volumeName="kubernetes.io/projected/2f1f5ba6-ccc5-4f0d-a135-743747ce667a-kube-api-access-z294f" seLinuxMountContext="" Nov 03 20:39:24.867083 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.866617 4305 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="747345eb-a407-423b-92df-c5d765a7da2d" volumeName="kubernetes.io/projected/747345eb-a407-423b-92df-c5d765a7da2d-kube-api-access-l2brl" seLinuxMountContext="" Nov 03 20:39:24.867083 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.866628 4305 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ced550a-b3cb-4605-b8dc-aab3b55804b4" volumeName="kubernetes.io/configmap/7ced550a-b3cb-4605-b8dc-aab3b55804b4-serviceca" seLinuxMountContext="" Nov 03 20:39:24.867083 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.866591 4305 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 20:39:24.867083 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.866639 4305 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bafd96d7-6982-4c93-81fc-d61643002fd1" volumeName="kubernetes.io/projected/bafd96d7-6982-4c93-81fc-d61643002fd1-kube-api-access-25sxl" seLinuxMountContext="" Nov 03 20:39:24.867083 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.866649 4305 factory.go:55] Registering systemd factory Nov 03 20:39:24.867083 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.866657 4305 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12658495-b5af-4c01-9c72-19b1173f7529" volumeName="kubernetes.io/configmap/12658495-b5af-4c01-9c72-19b1173f7529-ovnkube-script-lib" seLinuxMountContext="" Nov 03 20:39:24.867083 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.866663 4305 factory.go:223] Registration of the systemd container factory successfully Nov 03 20:39:24.867083 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.866668 4305 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6d81822a-bda4-4f5e-a5f4-05e1606f22bf" volumeName="kubernetes.io/empty-dir/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-etc-tuned" seLinuxMountContext="" Nov 03 20:39:24.867083 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.866679 4305 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e89ebac-2191-4acc-b4d4-ff326b3b74ac" volumeName="kubernetes.io/configmap/6e89ebac-2191-4acc-b4d4-ff326b3b74ac-iptables-alerter-script" seLinuxMountContext="" Nov 03 20:39:24.867083 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.866690 4305 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ced550a-b3cb-4605-b8dc-aab3b55804b4" volumeName="kubernetes.io/projected/7ced550a-b3cb-4605-b8dc-aab3b55804b4-kube-api-access-t6rjq" seLinuxMountContext="" Nov 03 20:39:24.867083 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.866701 4305 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf15bf7c-07c6-4d82-987b-aa4a332d5ca9" volumeName="kubernetes.io/empty-dir/cf15bf7c-07c6-4d82-987b-aa4a332d5ca9-tmp-dir" seLinuxMountContext="" Nov 03 20:39:24.867083 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.866714 4305 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf15bf7c-07c6-4d82-987b-aa4a332d5ca9" volumeName="kubernetes.io/projected/cf15bf7c-07c6-4d82-987b-aa4a332d5ca9-kube-api-access-t8b6m" seLinuxMountContext="" Nov 03 20:39:24.867083 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.866727 4305 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12658495-b5af-4c01-9c72-19b1173f7529" volumeName="kubernetes.io/configmap/12658495-b5af-4c01-9c72-19b1173f7529-env-overrides" seLinuxMountContext="" Nov 03 20:39:24.867083 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.866742 4305 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2dfb2512-b394-4a37-a5ba-a783d4e46c46" volumeName="kubernetes.io/projected/2dfb2512-b394-4a37-a5ba-a783d4e46c46-kube-api-access-5rlc8" seLinuxMountContext="" Nov 03 20:39:24.867083 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.866754 4305 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f1f5ba6-ccc5-4f0d-a135-743747ce667a" volumeName="kubernetes.io/configmap/2f1f5ba6-ccc5-4f0d-a135-743747ce667a-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 03 20:39:24.867083 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.866765 4305 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e72f317-1042-40b0-b709-dc3d650754b5" volumeName="kubernetes.io/configmap/3e72f317-1042-40b0-b709-dc3d650754b5-konnectivity-ca" seLinuxMountContext="" Nov 03 20:39:24.867083 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.866779 4305 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4a5df9f4-aaaa-4441-a2e0-28744e197828" volumeName="kubernetes.io/configmap/4a5df9f4-aaaa-4441-a2e0-28744e197828-multus-daemon-config" seLinuxMountContext="" Nov 03 20:39:24.867083 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.866792 4305 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4a5df9f4-aaaa-4441-a2e0-28744e197828" volumeName="kubernetes.io/projected/4a5df9f4-aaaa-4441-a2e0-28744e197828-kube-api-access-z2vcp" seLinuxMountContext="" Nov 03 20:39:24.867083 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.866804 4305 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6d81822a-bda4-4f5e-a5f4-05e1606f22bf" volumeName="kubernetes.io/empty-dir/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-tmp" seLinuxMountContext="" Nov 03 20:39:24.867083 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.866816 4305 reconstruct.go:97] "Volume reconstruction finished" Nov 03 20:39:24.867083 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.866824 4305 reconciler.go:29] "Reconciler: start to sync state" Nov 03 20:39:24.867083 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.866901 4305 factory.go:153] Registering CRI-O factory Nov 03 20:39:24.867083 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.866914 4305 factory.go:223] Registration of the crio container factory successfully Nov 03 20:39:24.867083 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.866934 4305 factory.go:103] Registering Raw factory Nov 03 20:39:24.867083 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.866949 4305 manager.go:1199] Started watching for new ooms in manager Nov 03 20:39:24.868539 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.867279 4305 manager.go:319] Starting recovery of all containers Nov 03 20:39:24.885525 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.885491 4305 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 20:39:24.887245 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.887214 4305 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 20:39:24.887245 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.887243 4305 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 20:39:24.887385 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.887267 4305 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 20:39:24.887385 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:39:24.887306 4305 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 20:39:24.898623 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.898592 4305 generic.go:356] "Generic (PLEG): container finished" podID="c6e5ca9eba1ab368215b5511cd840a25" containerID="8d1cb529c8966a57b48d3f0e7691288895d6376fc4689119efa3515eaaea608a" exitCode=0 Nov 03 20:39:24.917476 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.917423 4305 generic.go:356] "Generic (PLEG): container finished" podID="2f1f5ba6-ccc5-4f0d-a135-743747ce667a" containerID="9901fb3edf2819a2fc642ce42002a80d440bb14c3b735f61c0b63307f8d28756" exitCode=0 Nov 03 20:39:24.917476 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.917474 4305 generic.go:356] "Generic (PLEG): container finished" podID="2f1f5ba6-ccc5-4f0d-a135-743747ce667a" containerID="04a3b107559ce81fa0c42ff33e1881cd58b33943b5e722ae931ab3d9c15fd440" exitCode=0 Nov 03 20:39:24.917476 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.917484 4305 generic.go:356] "Generic (PLEG): container finished" podID="2f1f5ba6-ccc5-4f0d-a135-743747ce667a" containerID="566bae121881cf64b970d56bfe6f11e3ee068e6e1aad25b901da6a04544dad07" exitCode=0 Nov 03 20:39:24.917660 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.917495 4305 generic.go:356] "Generic (PLEG): container finished" podID="2f1f5ba6-ccc5-4f0d-a135-743747ce667a" containerID="d6bae224e7649b543b1c87f0ac71981b8487450781ea430ee15dbc3f1f43145d" exitCode=0 Nov 03 20:39:24.917660 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.917501 4305 generic.go:356] "Generic (PLEG): container finished" podID="2f1f5ba6-ccc5-4f0d-a135-743747ce667a" containerID="7c9155ccdf102f26a2ebe2477003e400f826a3f09a3af906af3c47c013d53db3" exitCode=0 Nov 03 20:39:24.917660 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.917508 4305 generic.go:356] "Generic (PLEG): container finished" podID="2f1f5ba6-ccc5-4f0d-a135-743747ce667a" containerID="3f3a9d69b49c65f870f7de2826852bf1e1c69765b95d32cc848a28571f3ad64e" exitCode=0 Nov 03 20:39:24.929804 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.929779 4305 manager.go:324] Recovery completed Nov 03 20:39:24.941615 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.941594 4305 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:39:24.942953 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.942933 4305 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-175.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:39:24.943037 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.942969 4305 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-175.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:39:24.943037 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.942983 4305 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-175.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 20:39:24.944416 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.944402 4305 cpu_manager.go:226] "Starting" policy="none" Nov 03 20:39:24.944486 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.944416 4305 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:39:24.944531 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.944416 4305 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 20:39:24.944531 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.944525 4305 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 20:39:24.944674 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.944660 4305 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 03 20:39:24.944708 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.944674 4305 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 03 20:39:24.944708 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.944698 4305 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 03 20:39:24.944708 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.944703 4305 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 03 20:39:24.944708 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.944709 4305 policy_none.go:50] "Start" Nov 03 20:39:24.947492 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.944719 4305 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 20:39:24.947492 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.944729 4305 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 20:39:24.947492 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.944850 4305 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 03 20:39:24.947492 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.944857 4305 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 03 20:39:24.947492 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.944863 4305 policy_none.go:44] "Start" Nov 03 20:39:24.947492 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.945357 4305 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-175.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:39:24.947492 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.945390 4305 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-175.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:39:24.947492 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.945405 4305 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-175.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 20:39:24.948838 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.948806 4305 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 20:39:24.951354 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.951339 4305 manager.go:341] "Starting Device Plugin manager" Nov 03 20:39:24.951455 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:39:24.951370 4305 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 20:39:24.951455 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.951383 4305 server.go:87] "Starting device plugin registration server" Nov 03 20:39:24.951717 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.951703 4305 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 20:39:24.951793 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.951718 4305 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 20:39:24.951793 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.951784 4305 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 20:39:24.951921 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.951887 4305 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 20:39:24.952042 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.951926 4305 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 20:39:24.952042 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.951933 4305 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 20:39:24.952042 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.951963 4305 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T20:39:24.951916542Z","UUID":"7a2aa7f8-9b38-440c-b7d8-4cc38d1e1aa6","Handler":null,"Name":"","Endpoint":""} Nov 03 20:39:24.953536 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.953511 4305 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 20:39:24.953632 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.953549 4305 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 20:39:24.953703 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:39:24.953683 4305 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 20:39:24.953755 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:39:24.953744 4305 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-175.us-west-1.compute.internal\" not found" Nov 03 20:39:24.988801 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.988240 4305 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-175.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-175.us-west-1.compute.internal"] Nov 03 20:39:24.989258 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.989232 4305 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:39:24.991852 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.991821 4305 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-175.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:39:24.991951 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.991867 4305 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-175.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:39:24.991951 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.991878 4305 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-175.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 20:39:24.992092 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.992081 4305 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:39:24.992222 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.992208 4305 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:39:24.993106 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.993083 4305 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-175.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:39:24.993214 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.993119 4305 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-175.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:39:24.993214 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.993133 4305 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-175.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 20:39:24.993214 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.993083 4305 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-175.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:39:24.993214 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.993192 4305 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-175.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:39:24.993214 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.993208 4305 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-175.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 20:39:24.993388 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:39:24.993302 4305 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-175.us-west-1.compute.internal\" not found" node="ip-10-0-128-175.us-west-1.compute.internal" Nov 03 20:39:24.993472 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.993421 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-175.us-west-1.compute.internal" event={"ID":"6873aa06186bc61b65aa0fa7f8c6d9a2","Type":"ContainerStarted","Data":"636a06c7fdd51b61647d612f2378b77fc865760a5467edc80bdf3c1249d19fa1"} Nov 03 20:39:24.993511 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.993482 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-175.us-west-1.compute.internal" event={"ID":"6873aa06186bc61b65aa0fa7f8c6d9a2","Type":"ContainerStarted","Data":"fc504e8ae03bf81d454277ab0a6d074765ef9f32dcb03c2bb8d8b38f2e759ba3"} Nov 03 20:39:24.993511 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.993508 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-175.us-west-1.compute.internal" event={"ID":"c6e5ca9eba1ab368215b5511cd840a25","Type":"ContainerStarted","Data":"3d0abf5deabbd92b2980052e594c39549ce5f0be38e3f75fe387585770e395bb"} Nov 03 20:39:24.993568 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.993522 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-175.us-west-1.compute.internal" event={"ID":"c6e5ca9eba1ab368215b5511cd840a25","Type":"ContainerDied","Data":"8d1cb529c8966a57b48d3f0e7691288895d6376fc4689119efa3515eaaea608a"} Nov 03 20:39:24.993568 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.993524 4305 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:39:24.993568 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.993535 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-175.us-west-1.compute.internal" event={"ID":"c6e5ca9eba1ab368215b5511cd840a25","Type":"ContainerStarted","Data":"90c4be718888538813c8970ee006a8aacbc94a7b254ed0aa3bc6b4bdcb72b2be"} Nov 03 20:39:24.994480 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.994467 4305 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-175.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:39:24.994530 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.994494 4305 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-175.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:39:24.994530 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:24.994506 4305 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-175.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 20:39:24.994639 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:39:24.994628 4305 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-175.us-west-1.compute.internal\" not found" node="ip-10-0-128-175.us-west-1.compute.internal" Nov 03 20:39:25.052857 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:25.052811 4305 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:39:25.054594 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:25.054578 4305 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-175.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:39:25.054690 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:25.054607 4305 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-175.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:39:25.054690 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:25.054620 4305 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-175.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 20:39:25.054690 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:25.054640 4305 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-175.us-west-1.compute.internal" Nov 03 20:39:25.063300 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:25.063271 4305 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:39:25.063479 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:25.063464 4305 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 03 20:39:25.111275 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:25.111242 4305 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:39:25.111405 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:25.111354 4305 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:39:25.113254 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:25.113225 4305 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:39:25.161715 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:25.161655 4305 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-175.us-west-1.compute.internal" Nov 03 20:39:25.163936 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:25.163916 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c6e5ca9eba1ab368215b5511cd840a25-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-175.us-west-1.compute.internal\" (UID: \"c6e5ca9eba1ab368215b5511cd840a25\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-175.us-west-1.compute.internal" Nov 03 20:39:25.164014 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:25.163950 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6e5ca9eba1ab368215b5511cd840a25-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-175.us-west-1.compute.internal\" (UID: \"c6e5ca9eba1ab368215b5511cd840a25\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-175.us-west-1.compute.internal" Nov 03 20:39:25.164014 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:25.163970 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6873aa06186bc61b65aa0fa7f8c6d9a2-config\") pod \"kube-apiserver-proxy-ip-10-0-128-175.us-west-1.compute.internal\" (UID: \"6873aa06186bc61b65aa0fa7f8c6d9a2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-175.us-west-1.compute.internal" Nov 03 20:39:25.212899 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:25.212858 4305 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-128-175.us-west-1.compute.internal" Nov 03 20:39:25.213041 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:25.212936 4305 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-175.us-west-1.compute.internal" Nov 03 20:39:25.213554 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:25.213528 4305 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-175.us-west-1.compute.internal" event="NodeNotReady" Nov 03 20:39:25.213691 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:25.213568 4305 setters.go:546] "Node became not ready" node="ip-10-0-128-175.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-03T20:39:25Z","lastTransitionTime":"2026-11-03T20:39:25Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Nov 03 20:39:25.213744 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:25.213727 4305 nodeinfomanager.go:471] existing CSINode "ip-10-0-128-175.us-west-1.compute.internal" is owned by different node (oldNodeID="69a0a1d3-de81-481c-8a44-630edf3e1781", newNodeID=""), cleaning up... Nov 03 20:39:25.224152 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:25.224127 4305 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 20:39:25.224249 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:39:25.224175 4305 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-175.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-175.us-west-1.compute.internal" Nov 03 20:39:25.224249 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:25.224188 4305 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-175.us-west-1.compute.internal" Nov 03 20:39:25.264672 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:25.264590 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c6e5ca9eba1ab368215b5511cd840a25-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-175.us-west-1.compute.internal\" (UID: \"c6e5ca9eba1ab368215b5511cd840a25\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-175.us-west-1.compute.internal" Nov 03 20:39:25.264672 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:25.264632 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6e5ca9eba1ab368215b5511cd840a25-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-175.us-west-1.compute.internal\" (UID: \"c6e5ca9eba1ab368215b5511cd840a25\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-175.us-west-1.compute.internal" Nov 03 20:39:25.264672 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:25.264653 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6873aa06186bc61b65aa0fa7f8c6d9a2-config\") pod \"kube-apiserver-proxy-ip-10-0-128-175.us-west-1.compute.internal\" (UID: \"6873aa06186bc61b65aa0fa7f8c6d9a2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-175.us-west-1.compute.internal" Nov 03 20:39:25.264886 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:25.264699 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6873aa06186bc61b65aa0fa7f8c6d9a2-config\") pod \"kube-apiserver-proxy-ip-10-0-128-175.us-west-1.compute.internal\" (UID: \"6873aa06186bc61b65aa0fa7f8c6d9a2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-175.us-west-1.compute.internal" Nov 03 20:39:25.264886 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:25.264710 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6e5ca9eba1ab368215b5511cd840a25-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-175.us-west-1.compute.internal\" (UID: \"c6e5ca9eba1ab368215b5511cd840a25\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-175.us-west-1.compute.internal" Nov 03 20:39:25.264886 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:25.264710 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c6e5ca9eba1ab368215b5511cd840a25-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-175.us-west-1.compute.internal\" (UID: \"c6e5ca9eba1ab368215b5511cd840a25\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-175.us-west-1.compute.internal" Nov 03 20:39:25.320465 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:25.320417 4305 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 20:39:25.320627 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:39:25.320488 4305 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-175.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-175.us-west-1.compute.internal" Nov 03 20:39:25.850250 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:25.850219 4305 apiserver.go:52] "Watching apiserver" Nov 03 20:39:25.920321 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:25.920282 4305 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-175.us-west-1.compute.internal" Nov 03 20:39:25.920484 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:25.920375 4305 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-175.us-west-1.compute.internal" Nov 03 20:39:25.981097 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:25.981065 4305 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 20:39:25.981259 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:39:25.981118 4305 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-175.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-175.us-west-1.compute.internal" Nov 03 20:39:25.981382 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:25.981366 4305 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 20:39:25.981425 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:39:25.981400 4305 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-175.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-175.us-west-1.compute.internal" Nov 03 20:39:26.036733 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.036710 4305 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 20:39:26.037078 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.037061 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-128-175.us-west-1.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq7n8","openshift-cluster-node-tuning-operator/tuned-6jqz7","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-175.us-west-1.compute.internal","openshift-multus/multus-4k6r2","openshift-multus/multus-additional-cni-plugins-shzwn","kube-system/konnectivity-agent-jvtt8","openshift-dns/node-resolver-gkm7b","openshift-image-registry/node-ca-p6hwx","openshift-network-diagnostics/network-check-target-hc89p","openshift-network-operator/iptables-alerter-lvcbj","kube-system/global-pull-secret-syncer-g4892","openshift-multus/network-metrics-daemon-xt5qx","openshift-ovn-kubernetes/ovnkube-node-4ljxs"] Nov 03 20:39:26.042659 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.042631 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xt5qx" Nov 03 20:39:26.061217 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.061189 4305 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 20:39:26.068648 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.068625 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/12658495-b5af-4c01-9c72-19b1173f7529-ovnkube-config\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:39:26.068723 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.068656 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f1f5ba6-ccc5-4f0d-a135-743747ce667a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-shzwn\" (UID: \"2f1f5ba6-ccc5-4f0d-a135-743747ce667a\") " pod="openshift-multus/multus-additional-cni-plugins-shzwn" Nov 03 20:39:26.068723 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.068674 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bafd96d7-6982-4c93-81fc-d61643002fd1-etc-selinux\") pod \"aws-ebs-csi-driver-node-jq7n8\" (UID: \"bafd96d7-6982-4c93-81fc-d61643002fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq7n8" Nov 03 20:39:26.068723 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.068691 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-host\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:39:26.068723 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.068707 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-host-var-lib-cni-bin\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:39:26.068840 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.068823 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z294f\" (UniqueName: \"kubernetes.io/projected/2f1f5ba6-ccc5-4f0d-a135-743747ce667a-kube-api-access-z294f\") pod \"multus-additional-cni-plugins-shzwn\" (UID: \"2f1f5ba6-ccc5-4f0d-a135-743747ce667a\") " pod="openshift-multus/multus-additional-cni-plugins-shzwn" Nov 03 20:39:26.068876 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.068859 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6e89ebac-2191-4acc-b4d4-ff326b3b74ac-iptables-alerter-script\") pod \"iptables-alerter-lvcbj\" (UID: \"6e89ebac-2191-4acc-b4d4-ff326b3b74ac\") " pod="openshift-network-operator/iptables-alerter-lvcbj" Nov 03 20:39:26.068906 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.068892 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/747345eb-a407-423b-92df-c5d765a7da2d-metrics-certs\") pod \"network-metrics-daemon-xt5qx\" (UID: \"747345eb-a407-423b-92df-c5d765a7da2d\") " pod="openshift-multus/network-metrics-daemon-xt5qx" Nov 03 20:39:26.068937 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.068916 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-etc-sysconfig\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:39:26.068974 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.068956 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-multus-conf-dir\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:39:26.069003 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.068996 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-etc-kubernetes\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:39:26.069041 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.069022 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/12658495-b5af-4c01-9c72-19b1173f7529-env-overrides\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:39:26.069075 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.069068 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f1f5ba6-ccc5-4f0d-a135-743747ce667a-cni-binary-copy\") pod \"multus-additional-cni-plugins-shzwn\" (UID: \"2f1f5ba6-ccc5-4f0d-a135-743747ce667a\") " pod="openshift-multus/multus-additional-cni-plugins-shzwn" Nov 03 20:39:26.069122 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.069106 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cf15bf7c-07c6-4d82-987b-aa4a332d5ca9-hosts-file\") pod \"node-resolver-gkm7b\" (UID: \"cf15bf7c-07c6-4d82-987b-aa4a332d5ca9\") " pod="openshift-dns/node-resolver-gkm7b" Nov 03 20:39:26.069154 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.069132 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bafd96d7-6982-4c93-81fc-d61643002fd1-device-dir\") pod \"aws-ebs-csi-driver-node-jq7n8\" (UID: \"bafd96d7-6982-4c93-81fc-d61643002fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq7n8" Nov 03 20:39:26.069154 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.069149 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25sxl\" (UniqueName: \"kubernetes.io/projected/bafd96d7-6982-4c93-81fc-d61643002fd1-kube-api-access-25sxl\") pod \"aws-ebs-csi-driver-node-jq7n8\" (UID: \"bafd96d7-6982-4c93-81fc-d61643002fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq7n8" Nov 03 20:39:26.069206 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.069167 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-etc-tuned\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:39:26.069206 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.069197 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmbq4\" (UniqueName: \"kubernetes.io/projected/6e89ebac-2191-4acc-b4d4-ff326b3b74ac-kube-api-access-tmbq4\") pod \"iptables-alerter-lvcbj\" (UID: \"6e89ebac-2191-4acc-b4d4-ff326b3b74ac\") " pod="openshift-network-operator/iptables-alerter-lvcbj" Nov 03 20:39:26.069263 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.069223 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-node-log\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:39:26.069263 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.069243 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-etc-tuned\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:39:26.069263 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.069247 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-host-run-k8s-cni-cncf-io\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:39:26.069385 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.069274 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-run-ovn\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:39:26.069385 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.069310 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/12658495-b5af-4c01-9c72-19b1173f7529-ovn-node-metrics-cert\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:39:26.069385 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.069356 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bafd96d7-6982-4c93-81fc-d61643002fd1-registration-dir\") pod \"aws-ebs-csi-driver-node-jq7n8\" (UID: \"bafd96d7-6982-4c93-81fc-d61643002fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq7n8" Nov 03 20:39:26.069507 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.069449 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-etc-sysctl-conf\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:39:26.069507 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.069476 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-multus-cni-dir\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:39:26.069507 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.069492 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-cnibin\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:39:26.069627 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.069511 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2f1f5ba6-ccc5-4f0d-a135-743747ce667a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-shzwn\" (UID: \"2f1f5ba6-ccc5-4f0d-a135-743747ce667a\") " pod="openshift-multus/multus-additional-cni-plugins-shzwn" Nov 03 20:39:26.069627 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.069529 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-system-cni-dir\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:39:26.069627 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.069547 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-host-run-netns\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:39:26.069627 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.069575 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-host-var-lib-kubelet\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:39:26.069627 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.069617 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-etc-openvswitch\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:39:26.069797 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.069665 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2f1f5ba6-ccc5-4f0d-a135-743747ce667a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-shzwn\" (UID: \"2f1f5ba6-ccc5-4f0d-a135-743747ce667a\") " pod="openshift-multus/multus-additional-cni-plugins-shzwn" Nov 03 20:39:26.069797 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.069687 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6e89ebac-2191-4acc-b4d4-ff326b3b74ac-host-slash\") pod \"iptables-alerter-lvcbj\" (UID: \"6e89ebac-2191-4acc-b4d4-ff326b3b74ac\") " pod="openshift-network-operator/iptables-alerter-lvcbj" Nov 03 20:39:26.069797 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.069704 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-log-socket\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:39:26.069797 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.069719 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj57q\" (UniqueName: \"kubernetes.io/projected/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-kube-api-access-wj57q\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:39:26.069797 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.069766 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-multus-socket-dir-parent\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:39:26.069997 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.069871 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-host-kubelet\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:39:26.069997 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.069911 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-host-cni-bin\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:39:26.069997 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.069945 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:39:26.070123 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.069994 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7ced550a-b3cb-4605-b8dc-aab3b55804b4-serviceca\") pod \"node-ca-p6hwx\" (UID: \"7ced550a-b3cb-4605-b8dc-aab3b55804b4\") " pod="openshift-image-registry/node-ca-p6hwx" Nov 03 20:39:26.070123 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.070021 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bafd96d7-6982-4c93-81fc-d61643002fd1-sys-fs\") pod \"aws-ebs-csi-driver-node-jq7n8\" (UID: \"bafd96d7-6982-4c93-81fc-d61643002fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq7n8" Nov 03 20:39:26.070123 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.070038 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-run\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:39:26.070123 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.070052 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-sys\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:39:26.070123 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.070089 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-host-var-lib-cni-multus\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:39:26.070292 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.070138 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/03985fe9-0dfe-4f9a-97db-bb4c0298be5c-kubelet-config\") pod \"global-pull-secret-syncer-g4892\" (UID: \"03985fe9-0dfe-4f9a-97db-bb4c0298be5c\") " pod="kube-system/global-pull-secret-syncer-g4892" Nov 03 20:39:26.070292 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.070158 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7ced550a-b3cb-4605-b8dc-aab3b55804b4-host\") pod \"node-ca-p6hwx\" (UID: \"7ced550a-b3cb-4605-b8dc-aab3b55804b4\") " pod="openshift-image-registry/node-ca-p6hwx" Nov 03 20:39:26.070292 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.070173 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-etc-kubernetes\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:39:26.070292 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.070191 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rlc8\" (UniqueName: \"kubernetes.io/projected/2dfb2512-b394-4a37-a5ba-a783d4e46c46-kube-api-access-5rlc8\") pod \"network-check-target-hc89p\" (UID: \"2dfb2512-b394-4a37-a5ba-a783d4e46c46\") " pod="openshift-network-diagnostics/network-check-target-hc89p" Nov 03 20:39:26.070292 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.070207 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/12658495-b5af-4c01-9c72-19b1173f7529-ovnkube-script-lib\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:39:26.070292 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.070267 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f1f5ba6-ccc5-4f0d-a135-743747ce667a-cnibin\") pod \"multus-additional-cni-plugins-shzwn\" (UID: \"2f1f5ba6-ccc5-4f0d-a135-743747ce667a\") " pod="openshift-multus/multus-additional-cni-plugins-shzwn" Nov 03 20:39:26.070292 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.070287 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bafd96d7-6982-4c93-81fc-d61643002fd1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jq7n8\" (UID: \"bafd96d7-6982-4c93-81fc-d61643002fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq7n8" Nov 03 20:39:26.070616 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.070304 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bafd96d7-6982-4c93-81fc-d61643002fd1-socket-dir\") pod \"aws-ebs-csi-driver-node-jq7n8\" (UID: \"bafd96d7-6982-4c93-81fc-d61643002fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq7n8" Nov 03 20:39:26.070616 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.070330 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-tmp\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:39:26.070616 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.070352 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-host-run-multus-certs\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:39:26.070616 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.070390 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-tmp\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:39:26.070616 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.070395 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2vcp\" (UniqueName: \"kubernetes.io/projected/4a5df9f4-aaaa-4441-a2e0-28744e197828-kube-api-access-z2vcp\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:39:26.070616 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.070454 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03985fe9-0dfe-4f9a-97db-bb4c0298be5c-original-pull-secret\") pod \"global-pull-secret-syncer-g4892\" (UID: \"03985fe9-0dfe-4f9a-97db-bb4c0298be5c\") " pod="kube-system/global-pull-secret-syncer-g4892" Nov 03 20:39:26.070616 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.070473 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f1f5ba6-ccc5-4f0d-a135-743747ce667a-system-cni-dir\") pod \"multus-additional-cni-plugins-shzwn\" (UID: \"2f1f5ba6-ccc5-4f0d-a135-743747ce667a\") " pod="openshift-multus/multus-additional-cni-plugins-shzwn" Nov 03 20:39:26.070616 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.070490 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-host-run-ovn-kubernetes\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:39:26.070616 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.070569 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-lib-modules\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:39:26.070616 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.070599 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-var-lib-kubelet\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:39:26.070984 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.070623 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-hostroot\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:39:26.070984 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.070659 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/03985fe9-0dfe-4f9a-97db-bb4c0298be5c-dbus\") pod \"global-pull-secret-syncer-g4892\" (UID: \"03985fe9-0dfe-4f9a-97db-bb4c0298be5c\") " pod="kube-system/global-pull-secret-syncer-g4892" Nov 03 20:39:26.070984 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.070682 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-host-run-netns\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:39:26.070984 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.070697 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-run-openvswitch\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:39:26.070984 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.070714 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cscp\" (UniqueName: \"kubernetes.io/projected/12658495-b5af-4c01-9c72-19b1173f7529-kube-api-access-6cscp\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:39:26.070984 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.070747 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-etc-modprobe-d\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:39:26.070984 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.070773 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-etc-systemd\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:39:26.070984 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.070798 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4a5df9f4-aaaa-4441-a2e0-28744e197828-cni-binary-copy\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:39:26.070984 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.070817 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-systemd-units\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:39:26.070984 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.070881 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-host-slash\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:39:26.070984 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.070911 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3e72f317-1042-40b0-b709-dc3d650754b5-konnectivity-ca\") pod \"konnectivity-agent-jvtt8\" (UID: \"3e72f317-1042-40b0-b709-dc3d650754b5\") " pod="kube-system/konnectivity-agent-jvtt8" Nov 03 20:39:26.070984 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.070936 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf15bf7c-07c6-4d82-987b-aa4a332d5ca9-tmp-dir\") pod \"node-resolver-gkm7b\" (UID: \"cf15bf7c-07c6-4d82-987b-aa4a332d5ca9\") " pod="openshift-dns/node-resolver-gkm7b" Nov 03 20:39:26.070984 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.070976 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8b6m\" (UniqueName: \"kubernetes.io/projected/cf15bf7c-07c6-4d82-987b-aa4a332d5ca9-kube-api-access-t8b6m\") pod \"node-resolver-gkm7b\" (UID: \"cf15bf7c-07c6-4d82-987b-aa4a332d5ca9\") " pod="openshift-dns/node-resolver-gkm7b" Nov 03 20:39:26.071526 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.070993 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-run-systemd\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:39:26.071526 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.071015 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-host-cni-netd\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:39:26.071526 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.071030 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3e72f317-1042-40b0-b709-dc3d650754b5-agent-certs\") pod \"konnectivity-agent-jvtt8\" (UID: \"3e72f317-1042-40b0-b709-dc3d650754b5\") " pod="kube-system/konnectivity-agent-jvtt8" Nov 03 20:39:26.071526 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.071062 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf15bf7c-07c6-4d82-987b-aa4a332d5ca9-tmp-dir\") pod \"node-resolver-gkm7b\" (UID: \"cf15bf7c-07c6-4d82-987b-aa4a332d5ca9\") " pod="openshift-dns/node-resolver-gkm7b" Nov 03 20:39:26.071526 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.071121 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6rjq\" (UniqueName: \"kubernetes.io/projected/7ced550a-b3cb-4605-b8dc-aab3b55804b4-kube-api-access-t6rjq\") pod \"node-ca-p6hwx\" (UID: \"7ced550a-b3cb-4605-b8dc-aab3b55804b4\") " pod="openshift-image-registry/node-ca-p6hwx" Nov 03 20:39:26.071526 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.071140 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2brl\" (UniqueName: \"kubernetes.io/projected/747345eb-a407-423b-92df-c5d765a7da2d-kube-api-access-l2brl\") pod \"network-metrics-daemon-xt5qx\" (UID: \"747345eb-a407-423b-92df-c5d765a7da2d\") " pod="openshift-multus/network-metrics-daemon-xt5qx" Nov 03 20:39:26.071526 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.071215 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-etc-sysctl-d\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:39:26.071526 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.071359 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4a5df9f4-aaaa-4441-a2e0-28744e197828-multus-daemon-config\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:39:26.071526 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.071390 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-var-lib-openvswitch\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:39:26.095313 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.095279 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 20:39:26.095503 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.095459 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 20:39:26.095562 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.095511 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:39:26.095562 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.095524 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 20:39:26.095562 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.095515 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 20:39:26.095705 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.095580 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 20:39:26.096081 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.096057 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 20:39:26.096217 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.096085 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 20:39:26.096217 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.096104 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 20:39:26.096217 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.096121 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 20:39:26.096367 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.096256 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 20:39:26.096556 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.096539 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 20:39:26.096628 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.096558 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 20:39:26.096628 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.096577 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 20:39:26.096628 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.096614 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-kqp9s\"" Nov 03 20:39:26.096628 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.096624 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-29dtl\"" Nov 03 20:39:26.096780 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.096649 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 20:39:26.096780 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.096648 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 20:39:26.096780 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.096662 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 20:39:26.096780 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.096663 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 20:39:26.096780 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.096749 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 20:39:26.096780 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.096751 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 20:39:26.096780 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.096759 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 20:39:26.097039 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.096908 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 20:39:26.097189 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.097171 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-nnzhp\"" Nov 03 20:39:26.097270 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.097251 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-j2mnh\"" Nov 03 20:39:26.097317 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.097285 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 20:39:26.097382 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.097371 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 20:39:26.097424 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.097410 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zxrmv\"" Nov 03 20:39:26.097537 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.097523 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-r6k78\"" Nov 03 20:39:26.097608 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.097595 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dm4wl\"" Nov 03 20:39:26.097681 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.097669 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 20:39:26.097681 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.097679 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4xfms\"" Nov 03 20:39:26.097776 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.097719 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 20:39:26.097816 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.097772 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-czwwq\"" Nov 03 20:39:26.097847 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.097841 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 20:39:26.097895 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.097883 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 20:39:26.097963 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.097950 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8nf42\"" Nov 03 20:39:26.098002 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.097990 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:39:26.098045 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.098033 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-zvv9n\"" Nov 03 20:39:26.098114 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.098102 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 20:39:26.098190 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.098179 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mtgd7\"" Nov 03 20:39:26.099378 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.099362 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/12658495-b5af-4c01-9c72-19b1173f7529-env-overrides\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:39:26.099460 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.099395 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/12658495-b5af-4c01-9c72-19b1173f7529-ovnkube-config\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:39:26.099524 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.099492 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f1f5ba6-ccc5-4f0d-a135-743747ce667a-cni-binary-copy\") pod \"multus-additional-cni-plugins-shzwn\" (UID: \"2f1f5ba6-ccc5-4f0d-a135-743747ce667a\") " pod="openshift-multus/multus-additional-cni-plugins-shzwn" Nov 03 20:39:26.099524 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.099505 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6e89ebac-2191-4acc-b4d4-ff326b3b74ac-iptables-alerter-script\") pod \"iptables-alerter-lvcbj\" (UID: \"6e89ebac-2191-4acc-b4d4-ff326b3b74ac\") " pod="openshift-network-operator/iptables-alerter-lvcbj" Nov 03 20:39:26.099688 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.099674 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/12658495-b5af-4c01-9c72-19b1173f7529-ovn-node-metrics-cert\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:39:26.099825 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.099809 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2f1f5ba6-ccc5-4f0d-a135-743747ce667a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-shzwn\" (UID: \"2f1f5ba6-ccc5-4f0d-a135-743747ce667a\") " pod="openshift-multus/multus-additional-cni-plugins-shzwn" Nov 03 20:39:26.100632 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.100586 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7ced550a-b3cb-4605-b8dc-aab3b55804b4-serviceca\") pod \"node-ca-p6hwx\" (UID: \"7ced550a-b3cb-4605-b8dc-aab3b55804b4\") " pod="openshift-image-registry/node-ca-p6hwx" Nov 03 20:39:26.100632 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.100586 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2f1f5ba6-ccc5-4f0d-a135-743747ce667a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-shzwn\" (UID: \"2f1f5ba6-ccc5-4f0d-a135-743747ce667a\") " pod="openshift-multus/multus-additional-cni-plugins-shzwn" Nov 03 20:39:26.100632 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.100611 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/12658495-b5af-4c01-9c72-19b1173f7529-ovnkube-script-lib\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:39:26.100756 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.100743 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03985fe9-0dfe-4f9a-97db-bb4c0298be5c-original-pull-secret\") pod \"global-pull-secret-syncer-g4892\" (UID: \"03985fe9-0dfe-4f9a-97db-bb4c0298be5c\") " pod="kube-system/global-pull-secret-syncer-g4892" Nov 03 20:39:26.101766 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.101751 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4a5df9f4-aaaa-4441-a2e0-28744e197828-cni-binary-copy\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:39:26.101823 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.101807 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4a5df9f4-aaaa-4441-a2e0-28744e197828-multus-daemon-config\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:39:26.101860 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.101834 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3e72f317-1042-40b0-b709-dc3d650754b5-agent-certs\") pod \"konnectivity-agent-jvtt8\" (UID: \"3e72f317-1042-40b0-b709-dc3d650754b5\") " pod="kube-system/konnectivity-agent-jvtt8" Nov 03 20:39:26.102139 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.101771 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3e72f317-1042-40b0-b709-dc3d650754b5-konnectivity-ca\") pod \"konnectivity-agent-jvtt8\" (UID: \"3e72f317-1042-40b0-b709-dc3d650754b5\") " pod="kube-system/konnectivity-agent-jvtt8" Nov 03 20:39:26.141164 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.141132 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 20:39:26.149473 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.149422 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2vcp\" (UniqueName: \"kubernetes.io/projected/4a5df9f4-aaaa-4441-a2e0-28744e197828-kube-api-access-z2vcp\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:39:26.149626 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.149607 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2brl\" (UniqueName: \"kubernetes.io/projected/747345eb-a407-423b-92df-c5d765a7da2d-kube-api-access-l2brl\") pod \"network-metrics-daemon-xt5qx\" (UID: \"747345eb-a407-423b-92df-c5d765a7da2d\") " pod="openshift-multus/network-metrics-daemon-xt5qx" Nov 03 20:39:26.149747 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.149728 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rlc8\" (UniqueName: \"kubernetes.io/projected/2dfb2512-b394-4a37-a5ba-a783d4e46c46-kube-api-access-5rlc8\") pod \"network-check-target-hc89p\" (UID: \"2dfb2512-b394-4a37-a5ba-a783d4e46c46\") " pod="openshift-network-diagnostics/network-check-target-hc89p" Nov 03 20:39:26.154314 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.154285 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8b6m\" (UniqueName: \"kubernetes.io/projected/cf15bf7c-07c6-4d82-987b-aa4a332d5ca9-kube-api-access-t8b6m\") pod \"node-resolver-gkm7b\" (UID: \"cf15bf7c-07c6-4d82-987b-aa4a332d5ca9\") " pod="openshift-dns/node-resolver-gkm7b" Nov 03 20:39:26.154547 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.154529 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z294f\" (UniqueName: \"kubernetes.io/projected/2f1f5ba6-ccc5-4f0d-a135-743747ce667a-kube-api-access-z294f\") pod \"multus-additional-cni-plugins-shzwn\" (UID: \"2f1f5ba6-ccc5-4f0d-a135-743747ce667a\") " pod="openshift-multus/multus-additional-cni-plugins-shzwn" Nov 03 20:39:26.154656 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.154643 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25sxl\" (UniqueName: \"kubernetes.io/projected/bafd96d7-6982-4c93-81fc-d61643002fd1-kube-api-access-25sxl\") pod \"aws-ebs-csi-driver-node-jq7n8\" (UID: \"bafd96d7-6982-4c93-81fc-d61643002fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq7n8" Nov 03 20:39:26.154770 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.154754 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6rjq\" (UniqueName: \"kubernetes.io/projected/7ced550a-b3cb-4605-b8dc-aab3b55804b4-kube-api-access-t6rjq\") pod \"node-ca-p6hwx\" (UID: \"7ced550a-b3cb-4605-b8dc-aab3b55804b4\") " pod="openshift-image-registry/node-ca-p6hwx" Nov 03 20:39:26.154901 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.154885 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cscp\" (UniqueName: \"kubernetes.io/projected/12658495-b5af-4c01-9c72-19b1173f7529-kube-api-access-6cscp\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:39:26.155362 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.155346 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmbq4\" (UniqueName: \"kubernetes.io/projected/6e89ebac-2191-4acc-b4d4-ff326b3b74ac-kube-api-access-tmbq4\") pod \"iptables-alerter-lvcbj\" (UID: \"6e89ebac-2191-4acc-b4d4-ff326b3b74ac\") " pod="openshift-network-operator/iptables-alerter-lvcbj" Nov 03 20:39:26.159567 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.159546 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj57q\" (UniqueName: \"kubernetes.io/projected/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-kube-api-access-wj57q\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:39:26.172555 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.172526 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-etc-kubernetes\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:39:26.172555 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.172556 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f1f5ba6-ccc5-4f0d-a135-743747ce667a-cnibin\") pod \"multus-additional-cni-plugins-shzwn\" (UID: \"2f1f5ba6-ccc5-4f0d-a135-743747ce667a\") " pod="openshift-multus/multus-additional-cni-plugins-shzwn" Nov 03 20:39:26.172749 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.172573 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bafd96d7-6982-4c93-81fc-d61643002fd1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jq7n8\" (UID: \"bafd96d7-6982-4c93-81fc-d61643002fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq7n8" Nov 03 20:39:26.172749 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.172589 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bafd96d7-6982-4c93-81fc-d61643002fd1-socket-dir\") pod \"aws-ebs-csi-driver-node-jq7n8\" (UID: \"bafd96d7-6982-4c93-81fc-d61643002fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq7n8" Nov 03 20:39:26.172749 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.172605 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-host-run-multus-certs\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:39:26.172749 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.172622 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f1f5ba6-ccc5-4f0d-a135-743747ce667a-system-cni-dir\") pod \"multus-additional-cni-plugins-shzwn\" (UID: \"2f1f5ba6-ccc5-4f0d-a135-743747ce667a\") " pod="openshift-multus/multus-additional-cni-plugins-shzwn" Nov 03 20:39:26.172749 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.172639 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-host-run-ovn-kubernetes\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:39:26.172749 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.172657 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-lib-modules\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:39:26.172749 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.172658 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-etc-kubernetes\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:39:26.172749 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.172667 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bafd96d7-6982-4c93-81fc-d61643002fd1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jq7n8\" (UID: \"bafd96d7-6982-4c93-81fc-d61643002fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq7n8" Nov 03 20:39:26.172749 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.172658 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f1f5ba6-ccc5-4f0d-a135-743747ce667a-cnibin\") pod \"multus-additional-cni-plugins-shzwn\" (UID: \"2f1f5ba6-ccc5-4f0d-a135-743747ce667a\") " pod="openshift-multus/multus-additional-cni-plugins-shzwn" Nov 03 20:39:26.172749 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.172680 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-var-lib-kubelet\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:39:26.172749 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.172696 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-host-run-multus-certs\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:39:26.172749 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.172714 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-hostroot\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:39:26.172749 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.172722 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bafd96d7-6982-4c93-81fc-d61643002fd1-socket-dir\") pod \"aws-ebs-csi-driver-node-jq7n8\" (UID: \"bafd96d7-6982-4c93-81fc-d61643002fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq7n8" Nov 03 20:39:26.172749 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.172702 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f1f5ba6-ccc5-4f0d-a135-743747ce667a-system-cni-dir\") pod \"multus-additional-cni-plugins-shzwn\" (UID: \"2f1f5ba6-ccc5-4f0d-a135-743747ce667a\") " pod="openshift-multus/multus-additional-cni-plugins-shzwn" Nov 03 20:39:26.173264 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.172758 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-host-run-ovn-kubernetes\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:39:26.173264 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.172746 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-hostroot\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:39:26.173264 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.172783 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-lib-modules\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:39:26.173264 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.172771 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-var-lib-kubelet\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:39:26.173264 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.172789 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/03985fe9-0dfe-4f9a-97db-bb4c0298be5c-dbus\") pod \"global-pull-secret-syncer-g4892\" (UID: \"03985fe9-0dfe-4f9a-97db-bb4c0298be5c\") " pod="kube-system/global-pull-secret-syncer-g4892" Nov 03 20:39:26.173264 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.172823 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-host-run-netns\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:39:26.173264 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.172837 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/03985fe9-0dfe-4f9a-97db-bb4c0298be5c-dbus\") pod \"global-pull-secret-syncer-g4892\" (UID: \"03985fe9-0dfe-4f9a-97db-bb4c0298be5c\") " pod="kube-system/global-pull-secret-syncer-g4892" Nov 03 20:39:26.173264 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.172854 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-run-openvswitch\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:39:26.173264 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.172945 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-run-openvswitch\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:39:26.173264 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.172947 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-host-run-netns\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:39:26.173264 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.172959 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-etc-modprobe-d\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:39:26.173264 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.172978 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-etc-systemd\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:39:26.173264 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.173007 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-systemd-units\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:39:26.173264 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.173022 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-host-slash\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:39:26.173264 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.173035 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-etc-modprobe-d\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:39:26.173264 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.173045 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-run-systemd\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:39:26.173264 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.173060 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-systemd-units\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:39:26.173264 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.173074 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-host-cni-netd\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:39:26.173264 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.173090 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-etc-systemd\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:39:26.173264 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.173099 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-etc-sysctl-d\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:39:26.173264 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.173112 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-host-slash\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:39:26.173264 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.173118 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-host-cni-netd\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:39:26.173264 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.173126 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-run-systemd\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:39:26.173264 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.173125 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-var-lib-openvswitch\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:39:26.173264 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.173170 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-var-lib-openvswitch\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:39:26.173264 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.173174 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-etc-sysctl-d\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:39:26.173264 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.173183 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f1f5ba6-ccc5-4f0d-a135-743747ce667a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-shzwn\" (UID: \"2f1f5ba6-ccc5-4f0d-a135-743747ce667a\") " pod="openshift-multus/multus-additional-cni-plugins-shzwn" Nov 03 20:39:26.173264 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.173212 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bafd96d7-6982-4c93-81fc-d61643002fd1-etc-selinux\") pod \"aws-ebs-csi-driver-node-jq7n8\" (UID: \"bafd96d7-6982-4c93-81fc-d61643002fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq7n8" Nov 03 20:39:26.173264 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.173234 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-host\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:39:26.173264 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.173280 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-host\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:39:26.173264 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.173282 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bafd96d7-6982-4c93-81fc-d61643002fd1-etc-selinux\") pod \"aws-ebs-csi-driver-node-jq7n8\" (UID: \"bafd96d7-6982-4c93-81fc-d61643002fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq7n8" Nov 03 20:39:26.173264 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.173281 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f1f5ba6-ccc5-4f0d-a135-743747ce667a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-shzwn\" (UID: \"2f1f5ba6-ccc5-4f0d-a135-743747ce667a\") " pod="openshift-multus/multus-additional-cni-plugins-shzwn" Nov 03 20:39:26.174890 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.173277 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-host-var-lib-cni-bin\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:39:26.174890 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.173309 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-host-var-lib-cni-bin\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:39:26.174890 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.173316 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/747345eb-a407-423b-92df-c5d765a7da2d-metrics-certs\") pod \"network-metrics-daemon-xt5qx\" (UID: \"747345eb-a407-423b-92df-c5d765a7da2d\") " pod="openshift-multus/network-metrics-daemon-xt5qx" Nov 03 20:39:26.174890 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.173341 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-etc-sysconfig\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:39:26.174890 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.173365 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-multus-conf-dir\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:39:26.174890 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.173389 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-etc-kubernetes\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:39:26.174890 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.173428 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cf15bf7c-07c6-4d82-987b-aa4a332d5ca9-hosts-file\") pod \"node-resolver-gkm7b\" (UID: \"cf15bf7c-07c6-4d82-987b-aa4a332d5ca9\") " pod="openshift-dns/node-resolver-gkm7b" Nov 03 20:39:26.174890 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.173470 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bafd96d7-6982-4c93-81fc-d61643002fd1-device-dir\") pod \"aws-ebs-csi-driver-node-jq7n8\" (UID: \"bafd96d7-6982-4c93-81fc-d61643002fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq7n8" Nov 03 20:39:26.174890 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.173479 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-etc-sysconfig\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:39:26.174890 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.173486 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-etc-kubernetes\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:39:26.174890 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.173496 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-node-log\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:39:26.174890 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.173479 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-multus-conf-dir\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:39:26.174890 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.173532 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cf15bf7c-07c6-4d82-987b-aa4a332d5ca9-hosts-file\") pod \"node-resolver-gkm7b\" (UID: \"cf15bf7c-07c6-4d82-987b-aa4a332d5ca9\") " pod="openshift-dns/node-resolver-gkm7b" Nov 03 20:39:26.174890 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.173535 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-node-log\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:39:26.174890 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.173566 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-host-run-k8s-cni-cncf-io\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:39:26.174890 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.173577 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bafd96d7-6982-4c93-81fc-d61643002fd1-device-dir\") pod \"aws-ebs-csi-driver-node-jq7n8\" (UID: \"bafd96d7-6982-4c93-81fc-d61643002fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq7n8" Nov 03 20:39:26.174890 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.173604 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-run-ovn\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:39:26.174890 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.173641 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-host-run-k8s-cni-cncf-io\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:39:26.174890 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.173680 4305 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 20:39:26.174890 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.173704 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bafd96d7-6982-4c93-81fc-d61643002fd1-registration-dir\") pod \"aws-ebs-csi-driver-node-jq7n8\" (UID: \"bafd96d7-6982-4c93-81fc-d61643002fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq7n8" Nov 03 20:39:26.174890 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.173742 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-run-ovn\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:39:26.174890 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.173750 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-etc-sysctl-conf\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:39:26.174890 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.173753 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bafd96d7-6982-4c93-81fc-d61643002fd1-registration-dir\") pod \"aws-ebs-csi-driver-node-jq7n8\" (UID: \"bafd96d7-6982-4c93-81fc-d61643002fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq7n8" Nov 03 20:39:26.174890 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.173809 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-multus-cni-dir\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:39:26.174890 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.173848 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-cnibin\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:39:26.174890 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.173877 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-etc-sysctl-conf\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:39:26.174890 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.173899 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-multus-cni-dir\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:39:26.174890 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.173882 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-cnibin\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:39:26.174890 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.173921 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-system-cni-dir\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:39:26.174890 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.173939 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-host-run-netns\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:39:26.174890 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.173963 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-host-var-lib-kubelet\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:39:26.174890 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.173998 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-host-run-netns\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:39:26.174890 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.173987 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-etc-openvswitch\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:39:26.174890 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.174019 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-host-var-lib-kubelet\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:39:26.174890 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.174028 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-etc-openvswitch\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:39:26.174890 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.174042 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-system-cni-dir\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:39:26.174890 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.174035 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6e89ebac-2191-4acc-b4d4-ff326b3b74ac-host-slash\") pod \"iptables-alerter-lvcbj\" (UID: \"6e89ebac-2191-4acc-b4d4-ff326b3b74ac\") " pod="openshift-network-operator/iptables-alerter-lvcbj" Nov 03 20:39:26.174890 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.174077 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-log-socket\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:39:26.174890 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.174081 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6e89ebac-2191-4acc-b4d4-ff326b3b74ac-host-slash\") pod \"iptables-alerter-lvcbj\" (UID: \"6e89ebac-2191-4acc-b4d4-ff326b3b74ac\") " pod="openshift-network-operator/iptables-alerter-lvcbj" Nov 03 20:39:26.174890 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.174103 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-multus-socket-dir-parent\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:39:26.174890 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.174138 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-host-kubelet\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:39:26.174890 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.174133 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-log-socket\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:39:26.174890 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.174157 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-host-cni-bin\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:39:26.174890 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.174173 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-multus-socket-dir-parent\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:39:26.174890 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.174189 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-host-kubelet\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:39:26.174890 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.174204 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-host-cni-bin\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:39:26.174890 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.174183 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:39:26.174890 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.174242 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12658495-b5af-4c01-9c72-19b1173f7529-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4ljxs\" (UID: \"12658495-b5af-4c01-9c72-19b1173f7529\") " pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:39:26.174890 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.174251 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bafd96d7-6982-4c93-81fc-d61643002fd1-sys-fs\") pod \"aws-ebs-csi-driver-node-jq7n8\" (UID: \"bafd96d7-6982-4c93-81fc-d61643002fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq7n8" Nov 03 20:39:26.174890 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.174279 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-run\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:39:26.174890 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.174297 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-sys\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:39:26.174890 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.174313 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-host-var-lib-cni-multus\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:39:26.174890 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.174336 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/03985fe9-0dfe-4f9a-97db-bb4c0298be5c-kubelet-config\") pod \"global-pull-secret-syncer-g4892\" (UID: \"03985fe9-0dfe-4f9a-97db-bb4c0298be5c\") " pod="kube-system/global-pull-secret-syncer-g4892" Nov 03 20:39:26.174890 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.174346 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bafd96d7-6982-4c93-81fc-d61643002fd1-sys-fs\") pod \"aws-ebs-csi-driver-node-jq7n8\" (UID: \"bafd96d7-6982-4c93-81fc-d61643002fd1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jq7n8" Nov 03 20:39:26.174890 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.174371 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-run\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:39:26.174890 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.174379 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6d81822a-bda4-4f5e-a5f4-05e1606f22bf-sys\") pod \"tuned-6jqz7\" (UID: \"6d81822a-bda4-4f5e-a5f4-05e1606f22bf\") " pod="openshift-cluster-node-tuning-operator/tuned-6jqz7" Nov 03 20:39:26.174890 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.174390 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/03985fe9-0dfe-4f9a-97db-bb4c0298be5c-kubelet-config\") pod \"global-pull-secret-syncer-g4892\" (UID: \"03985fe9-0dfe-4f9a-97db-bb4c0298be5c\") " pod="kube-system/global-pull-secret-syncer-g4892" Nov 03 20:39:26.174890 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.174395 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7ced550a-b3cb-4605-b8dc-aab3b55804b4-host\") pod \"node-ca-p6hwx\" (UID: \"7ced550a-b3cb-4605-b8dc-aab3b55804b4\") " pod="openshift-image-registry/node-ca-p6hwx" Nov 03 20:39:26.174890 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.174391 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4a5df9f4-aaaa-4441-a2e0-28744e197828-host-var-lib-cni-multus\") pod \"multus-4k6r2\" (UID: \"4a5df9f4-aaaa-4441-a2e0-28744e197828\") " pod="openshift-multus/multus-4k6r2" Nov 03 20:39:26.174890 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.174480 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7ced550a-b3cb-4605-b8dc-aab3b55804b4-host\") pod \"node-ca-p6hwx\" (UID: \"7ced550a-b3cb-4605-b8dc-aab3b55804b4\") " pod="openshift-image-registry/node-ca-p6hwx" Nov 03 20:39:26.177041 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.176376 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/747345eb-a407-423b-92df-c5d765a7da2d-metrics-certs\") pod \"network-metrics-daemon-xt5qx\" (UID: \"747345eb-a407-423b-92df-c5d765a7da2d\") " pod="openshift-multus/network-metrics-daemon-xt5qx" Nov 03 20:39:26.343154 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.343114 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xt5qx" Nov 03 20:39:26.766189 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.766096 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xt5qx"] Nov 03 20:39:26.770349 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:26.770319 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod747345eb_a407_423b_92df_c5d765a7da2d.slice/crio-ce438e5c86235c7b1a7feec563523bb94086fa18f50bc925a70e4aa93c6282b9 WatchSource:0}: Error finding container ce438e5c86235c7b1a7feec563523bb94086fa18f50bc925a70e4aa93c6282b9: Status 404 returned error can't find the container with id ce438e5c86235c7b1a7feec563523bb94086fa18f50bc925a70e4aa93c6282b9 Nov 03 20:39:26.772801 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.772782 4305 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:39:26.923946 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:26.923907 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xt5qx" event={"ID":"747345eb-a407-423b-92df-c5d765a7da2d","Type":"ContainerStarted","Data":"ce438e5c86235c7b1a7feec563523bb94086fa18f50bc925a70e4aa93c6282b9"} Nov 03 20:39:28.931036 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:28.930991 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xt5qx" event={"ID":"747345eb-a407-423b-92df-c5d765a7da2d","Type":"ContainerStarted","Data":"ddd855c1adc3d2871329e99c2cebaf19559cd29f8a20431da89bcae6b16bca0f"} Nov 03 20:39:28.931421 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:28.931056 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xt5qx" event={"ID":"747345eb-a407-423b-92df-c5d765a7da2d","Type":"ContainerStarted","Data":"494c8fcf42e37e68b9f4489d36b72d3ab7e0fd7faea4afb68603b397feda66c2"} Nov 03 20:39:29.810604 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:29.810567 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-hc89p" Nov 03 20:39:29.942752 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:29.942719 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:39:34.113340 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:34.113310 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:39:34.113732 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:34.113562 4305 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 20:39:35.563110 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:35.563078 4305 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-175.us-west-1.compute.internal" event="NodeReady" Nov 03 20:39:36.228103 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:36.228077 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jvtt8" Nov 03 20:39:38.767820 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:38.767778 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-b4gn6"] Nov 03 20:39:38.773768 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:38.773737 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-b4gn6" Nov 03 20:39:38.809715 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:38.809686 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9k4g2"] Nov 03 20:39:38.811408 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:38.811393 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9k4g2" Nov 03 20:39:38.826411 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:38.826389 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 20:39:38.827381 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:38.827352 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9fj5k\"" Nov 03 20:39:38.827381 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:38.827369 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 20:39:38.827381 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:38.827377 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 20:39:38.845829 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:38.845793 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ffc31124-d81f-40af-876c-da6c6979714b-data-volume\") pod \"insights-runtime-extractor-b4gn6\" (UID: \"ffc31124-d81f-40af-876c-da6c6979714b\") " pod="openshift-insights/insights-runtime-extractor-b4gn6" Nov 03 20:39:38.845969 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:38.845835 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwlr9\" (UniqueName: \"kubernetes.io/projected/ffc31124-d81f-40af-876c-da6c6979714b-kube-api-access-hwlr9\") pod \"insights-runtime-extractor-b4gn6\" (UID: \"ffc31124-d81f-40af-876c-da6c6979714b\") " pod="openshift-insights/insights-runtime-extractor-b4gn6" Nov 03 20:39:38.845969 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:38.845859 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/750c5782-16b3-4ab6-a88d-cd2e326fcc07-metrics-tls\") pod \"dns-default-9k4g2\" (UID: \"750c5782-16b3-4ab6-a88d-cd2e326fcc07\") " pod="openshift-dns/dns-default-9k4g2" Nov 03 20:39:38.845969 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:38.845881 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/750c5782-16b3-4ab6-a88d-cd2e326fcc07-tmp-dir\") pod \"dns-default-9k4g2\" (UID: \"750c5782-16b3-4ab6-a88d-cd2e326fcc07\") " pod="openshift-dns/dns-default-9k4g2" Nov 03 20:39:38.846106 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:38.845979 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9clzf\" (UniqueName: \"kubernetes.io/projected/750c5782-16b3-4ab6-a88d-cd2e326fcc07-kube-api-access-9clzf\") pod \"dns-default-9k4g2\" (UID: \"750c5782-16b3-4ab6-a88d-cd2e326fcc07\") " pod="openshift-dns/dns-default-9k4g2" Nov 03 20:39:38.846106 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:38.846013 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ffc31124-d81f-40af-876c-da6c6979714b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b4gn6\" (UID: \"ffc31124-d81f-40af-876c-da6c6979714b\") " pod="openshift-insights/insights-runtime-extractor-b4gn6" Nov 03 20:39:38.846106 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:38.846037 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ffc31124-d81f-40af-876c-da6c6979714b-crio-socket\") pod \"insights-runtime-extractor-b4gn6\" (UID: \"ffc31124-d81f-40af-876c-da6c6979714b\") " pod="openshift-insights/insights-runtime-extractor-b4gn6" Nov 03 20:39:38.846106 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:38.846051 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/750c5782-16b3-4ab6-a88d-cd2e326fcc07-config-volume\") pod \"dns-default-9k4g2\" (UID: \"750c5782-16b3-4ab6-a88d-cd2e326fcc07\") " pod="openshift-dns/dns-default-9k4g2" Nov 03 20:39:38.846277 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:38.846150 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ffc31124-d81f-40af-876c-da6c6979714b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b4gn6\" (UID: \"ffc31124-d81f-40af-876c-da6c6979714b\") " pod="openshift-insights/insights-runtime-extractor-b4gn6" Nov 03 20:39:38.858523 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:38.858498 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 20:39:38.865671 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:38.865652 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 20:39:38.891817 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:38.891790 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-b4gn6"] Nov 03 20:39:38.915256 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:38.915226 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-wfv5k"] Nov 03 20:39:38.916998 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:38.916982 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-566dbff8bb-sf4f6"] Nov 03 20:39:38.917121 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:38.917106 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wfv5k" Nov 03 20:39:38.918604 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:38.918587 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-566dbff8bb-sf4f6" Nov 03 20:39:38.929051 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:38.928966 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 20:39:38.929344 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:38.929325 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9ll5z\"" Nov 03 20:39:38.947174 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:38.947152 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d2d9ad62-a00d-4564-b22b-78e932bfb42c-trusted-ca\") pod \"image-registry-566dbff8bb-sf4f6\" (UID: \"d2d9ad62-a00d-4564-b22b-78e932bfb42c\") " pod="openshift-image-registry/image-registry-566dbff8bb-sf4f6" Nov 03 20:39:38.947289 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:38.947181 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d2d9ad62-a00d-4564-b22b-78e932bfb42c-installation-pull-secrets\") pod \"image-registry-566dbff8bb-sf4f6\" (UID: \"d2d9ad62-a00d-4564-b22b-78e932bfb42c\") " pod="openshift-image-registry/image-registry-566dbff8bb-sf4f6" Nov 03 20:39:38.947289 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:38.947206 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ffc31124-d81f-40af-876c-da6c6979714b-crio-socket\") pod \"insights-runtime-extractor-b4gn6\" (UID: \"ffc31124-d81f-40af-876c-da6c6979714b\") " pod="openshift-insights/insights-runtime-extractor-b4gn6" Nov 03 20:39:38.947289 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:38.947227 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/750c5782-16b3-4ab6-a88d-cd2e326fcc07-config-volume\") pod \"dns-default-9k4g2\" (UID: \"750c5782-16b3-4ab6-a88d-cd2e326fcc07\") " pod="openshift-dns/dns-default-9k4g2" Nov 03 20:39:38.947289 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:38.947273 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d2d9ad62-a00d-4564-b22b-78e932bfb42c-ca-trust-extracted\") pod \"image-registry-566dbff8bb-sf4f6\" (UID: \"d2d9ad62-a00d-4564-b22b-78e932bfb42c\") " pod="openshift-image-registry/image-registry-566dbff8bb-sf4f6" Nov 03 20:39:38.947427 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:38.947339 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnlhc\" (UniqueName: \"kubernetes.io/projected/d2d9ad62-a00d-4564-b22b-78e932bfb42c-kube-api-access-xnlhc\") pod \"image-registry-566dbff8bb-sf4f6\" (UID: \"d2d9ad62-a00d-4564-b22b-78e932bfb42c\") " pod="openshift-image-registry/image-registry-566dbff8bb-sf4f6" Nov 03 20:39:38.947427 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:38.947363 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ffc31124-d81f-40af-876c-da6c6979714b-crio-socket\") pod \"insights-runtime-extractor-b4gn6\" (UID: \"ffc31124-d81f-40af-876c-da6c6979714b\") " pod="openshift-insights/insights-runtime-extractor-b4gn6" Nov 03 20:39:38.947427 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:38.947418 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ffc31124-d81f-40af-876c-da6c6979714b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b4gn6\" (UID: \"ffc31124-d81f-40af-876c-da6c6979714b\") " pod="openshift-insights/insights-runtime-extractor-b4gn6" Nov 03 20:39:38.947558 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:38.947512 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ffc31124-d81f-40af-876c-da6c6979714b-data-volume\") pod \"insights-runtime-extractor-b4gn6\" (UID: \"ffc31124-d81f-40af-876c-da6c6979714b\") " pod="openshift-insights/insights-runtime-extractor-b4gn6" Nov 03 20:39:38.947558 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:38.947538 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/10e7de28-0482-4218-85f6-0960071ee6f4-cert\") pod \"ingress-canary-wfv5k\" (UID: \"10e7de28-0482-4218-85f6-0960071ee6f4\") " pod="openshift-ingress-canary/ingress-canary-wfv5k" Nov 03 20:39:38.947630 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:38.947565 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwlr9\" (UniqueName: \"kubernetes.io/projected/ffc31124-d81f-40af-876c-da6c6979714b-kube-api-access-hwlr9\") pod \"insights-runtime-extractor-b4gn6\" (UID: \"ffc31124-d81f-40af-876c-da6c6979714b\") " pod="openshift-insights/insights-runtime-extractor-b4gn6" Nov 03 20:39:38.947681 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:38.947646 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/750c5782-16b3-4ab6-a88d-cd2e326fcc07-metrics-tls\") pod \"dns-default-9k4g2\" (UID: \"750c5782-16b3-4ab6-a88d-cd2e326fcc07\") " pod="openshift-dns/dns-default-9k4g2" Nov 03 20:39:38.947736 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:38.947684 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/750c5782-16b3-4ab6-a88d-cd2e326fcc07-tmp-dir\") pod \"dns-default-9k4g2\" (UID: \"750c5782-16b3-4ab6-a88d-cd2e326fcc07\") " pod="openshift-dns/dns-default-9k4g2" Nov 03 20:39:38.947736 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:38.947725 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d2d9ad62-a00d-4564-b22b-78e932bfb42c-bound-sa-token\") pod \"image-registry-566dbff8bb-sf4f6\" (UID: \"d2d9ad62-a00d-4564-b22b-78e932bfb42c\") " pod="openshift-image-registry/image-registry-566dbff8bb-sf4f6" Nov 03 20:39:38.947835 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:38.947757 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9clzf\" (UniqueName: \"kubernetes.io/projected/750c5782-16b3-4ab6-a88d-cd2e326fcc07-kube-api-access-9clzf\") pod \"dns-default-9k4g2\" (UID: \"750c5782-16b3-4ab6-a88d-cd2e326fcc07\") " pod="openshift-dns/dns-default-9k4g2" Nov 03 20:39:38.947835 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:38.947786 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d2d9ad62-a00d-4564-b22b-78e932bfb42c-registry-tls\") pod \"image-registry-566dbff8bb-sf4f6\" (UID: \"d2d9ad62-a00d-4564-b22b-78e932bfb42c\") " pod="openshift-image-registry/image-registry-566dbff8bb-sf4f6" Nov 03 20:39:38.947835 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:38.947794 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/750c5782-16b3-4ab6-a88d-cd2e326fcc07-config-volume\") pod \"dns-default-9k4g2\" (UID: \"750c5782-16b3-4ab6-a88d-cd2e326fcc07\") " pod="openshift-dns/dns-default-9k4g2" Nov 03 20:39:38.947835 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:38.947811 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwlhv\" (UniqueName: \"kubernetes.io/projected/10e7de28-0482-4218-85f6-0960071ee6f4-kube-api-access-fwlhv\") pod \"ingress-canary-wfv5k\" (UID: \"10e7de28-0482-4218-85f6-0960071ee6f4\") " pod="openshift-ingress-canary/ingress-canary-wfv5k" Nov 03 20:39:38.948027 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:38.947852 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ffc31124-d81f-40af-876c-da6c6979714b-data-volume\") pod \"insights-runtime-extractor-b4gn6\" (UID: \"ffc31124-d81f-40af-876c-da6c6979714b\") " pod="openshift-insights/insights-runtime-extractor-b4gn6" Nov 03 20:39:38.948027 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:38.947876 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d2d9ad62-a00d-4564-b22b-78e932bfb42c-image-registry-private-configuration\") pod \"image-registry-566dbff8bb-sf4f6\" (UID: \"d2d9ad62-a00d-4564-b22b-78e932bfb42c\") " pod="openshift-image-registry/image-registry-566dbff8bb-sf4f6" Nov 03 20:39:38.948027 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:38.947986 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ffc31124-d81f-40af-876c-da6c6979714b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b4gn6\" (UID: \"ffc31124-d81f-40af-876c-da6c6979714b\") " pod="openshift-insights/insights-runtime-extractor-b4gn6" Nov 03 20:39:38.948027 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:38.948016 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d2d9ad62-a00d-4564-b22b-78e932bfb42c-registry-certificates\") pod \"image-registry-566dbff8bb-sf4f6\" (UID: \"d2d9ad62-a00d-4564-b22b-78e932bfb42c\") " pod="openshift-image-registry/image-registry-566dbff8bb-sf4f6" Nov 03 20:39:38.948172 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:38.948065 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/750c5782-16b3-4ab6-a88d-cd2e326fcc07-tmp-dir\") pod \"dns-default-9k4g2\" (UID: \"750c5782-16b3-4ab6-a88d-cd2e326fcc07\") " pod="openshift-dns/dns-default-9k4g2" Nov 03 20:39:38.948399 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:38.948383 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ffc31124-d81f-40af-876c-da6c6979714b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b4gn6\" (UID: \"ffc31124-d81f-40af-876c-da6c6979714b\") " pod="openshift-insights/insights-runtime-extractor-b4gn6" Nov 03 20:39:38.949830 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:38.949805 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ffc31124-d81f-40af-876c-da6c6979714b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b4gn6\" (UID: \"ffc31124-d81f-40af-876c-da6c6979714b\") " pod="openshift-insights/insights-runtime-extractor-b4gn6" Nov 03 20:39:38.950020 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:38.950005 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/750c5782-16b3-4ab6-a88d-cd2e326fcc07-metrics-tls\") pod \"dns-default-9k4g2\" (UID: \"750c5782-16b3-4ab6-a88d-cd2e326fcc07\") " pod="openshift-dns/dns-default-9k4g2" Nov 03 20:39:38.962534 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:38.962504 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9k4g2"] Nov 03 20:39:38.972387 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:38.972366 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 20:39:38.972522 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:38.972480 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 20:39:39.024537 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:39.024425 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 20:39:39.025064 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:39.025041 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wfv5k"] Nov 03 20:39:39.025490 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:39.025472 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-566dbff8bb-sf4f6"] Nov 03 20:39:39.027009 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:39.026988 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwlr9\" (UniqueName: \"kubernetes.io/projected/ffc31124-d81f-40af-876c-da6c6979714b-kube-api-access-hwlr9\") pod \"insights-runtime-extractor-b4gn6\" (UID: \"ffc31124-d81f-40af-876c-da6c6979714b\") " pod="openshift-insights/insights-runtime-extractor-b4gn6" Nov 03 20:39:39.027149 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:39.027132 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9clzf\" (UniqueName: \"kubernetes.io/projected/750c5782-16b3-4ab6-a88d-cd2e326fcc07-kube-api-access-9clzf\") pod \"dns-default-9k4g2\" (UID: \"750c5782-16b3-4ab6-a88d-cd2e326fcc07\") " pod="openshift-dns/dns-default-9k4g2" Nov 03 20:39:39.028957 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:39.028941 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 20:39:39.029076 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:39.029060 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-fjpbj\"" Nov 03 20:39:39.029076 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:39.029069 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 20:39:39.029268 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:39.029253 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 20:39:39.029320 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:39.029308 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-plrnn\"" Nov 03 20:39:39.029370 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:39.029358 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 20:39:39.049175 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:39.049149 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d2d9ad62-a00d-4564-b22b-78e932bfb42c-trusted-ca\") pod \"image-registry-566dbff8bb-sf4f6\" (UID: \"d2d9ad62-a00d-4564-b22b-78e932bfb42c\") " pod="openshift-image-registry/image-registry-566dbff8bb-sf4f6" Nov 03 20:39:39.049345 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:39.049181 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d2d9ad62-a00d-4564-b22b-78e932bfb42c-installation-pull-secrets\") pod \"image-registry-566dbff8bb-sf4f6\" (UID: \"d2d9ad62-a00d-4564-b22b-78e932bfb42c\") " pod="openshift-image-registry/image-registry-566dbff8bb-sf4f6" Nov 03 20:39:39.049345 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:39.049200 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d2d9ad62-a00d-4564-b22b-78e932bfb42c-ca-trust-extracted\") pod \"image-registry-566dbff8bb-sf4f6\" (UID: \"d2d9ad62-a00d-4564-b22b-78e932bfb42c\") " pod="openshift-image-registry/image-registry-566dbff8bb-sf4f6" Nov 03 20:39:39.049345 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:39.049218 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnlhc\" (UniqueName: \"kubernetes.io/projected/d2d9ad62-a00d-4564-b22b-78e932bfb42c-kube-api-access-xnlhc\") pod \"image-registry-566dbff8bb-sf4f6\" (UID: \"d2d9ad62-a00d-4564-b22b-78e932bfb42c\") " pod="openshift-image-registry/image-registry-566dbff8bb-sf4f6" Nov 03 20:39:39.049345 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:39.049245 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/10e7de28-0482-4218-85f6-0960071ee6f4-cert\") pod \"ingress-canary-wfv5k\" (UID: \"10e7de28-0482-4218-85f6-0960071ee6f4\") " pod="openshift-ingress-canary/ingress-canary-wfv5k" Nov 03 20:39:39.049345 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:39.049268 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d2d9ad62-a00d-4564-b22b-78e932bfb42c-bound-sa-token\") pod \"image-registry-566dbff8bb-sf4f6\" (UID: \"d2d9ad62-a00d-4564-b22b-78e932bfb42c\") " pod="openshift-image-registry/image-registry-566dbff8bb-sf4f6" Nov 03 20:39:39.049345 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:39.049294 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d2d9ad62-a00d-4564-b22b-78e932bfb42c-registry-tls\") pod \"image-registry-566dbff8bb-sf4f6\" (UID: \"d2d9ad62-a00d-4564-b22b-78e932bfb42c\") " pod="openshift-image-registry/image-registry-566dbff8bb-sf4f6" Nov 03 20:39:39.049345 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:39.049322 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwlhv\" (UniqueName: \"kubernetes.io/projected/10e7de28-0482-4218-85f6-0960071ee6f4-kube-api-access-fwlhv\") pod \"ingress-canary-wfv5k\" (UID: \"10e7de28-0482-4218-85f6-0960071ee6f4\") " pod="openshift-ingress-canary/ingress-canary-wfv5k" Nov 03 20:39:39.049722 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:39.049352 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d2d9ad62-a00d-4564-b22b-78e932bfb42c-image-registry-private-configuration\") pod \"image-registry-566dbff8bb-sf4f6\" (UID: \"d2d9ad62-a00d-4564-b22b-78e932bfb42c\") " pod="openshift-image-registry/image-registry-566dbff8bb-sf4f6" Nov 03 20:39:39.049722 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:39.049392 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d2d9ad62-a00d-4564-b22b-78e932bfb42c-registry-certificates\") pod \"image-registry-566dbff8bb-sf4f6\" (UID: \"d2d9ad62-a00d-4564-b22b-78e932bfb42c\") " pod="openshift-image-registry/image-registry-566dbff8bb-sf4f6" Nov 03 20:39:39.049977 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:39.049936 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d2d9ad62-a00d-4564-b22b-78e932bfb42c-ca-trust-extracted\") pod \"image-registry-566dbff8bb-sf4f6\" (UID: \"d2d9ad62-a00d-4564-b22b-78e932bfb42c\") " pod="openshift-image-registry/image-registry-566dbff8bb-sf4f6" Nov 03 20:39:39.050396 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:39.050311 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d2d9ad62-a00d-4564-b22b-78e932bfb42c-registry-certificates\") pod \"image-registry-566dbff8bb-sf4f6\" (UID: \"d2d9ad62-a00d-4564-b22b-78e932bfb42c\") " pod="openshift-image-registry/image-registry-566dbff8bb-sf4f6" Nov 03 20:39:39.051099 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:39.051073 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d2d9ad62-a00d-4564-b22b-78e932bfb42c-trusted-ca\") pod \"image-registry-566dbff8bb-sf4f6\" (UID: \"d2d9ad62-a00d-4564-b22b-78e932bfb42c\") " pod="openshift-image-registry/image-registry-566dbff8bb-sf4f6" Nov 03 20:39:39.052139 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:39.052120 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d2d9ad62-a00d-4564-b22b-78e932bfb42c-registry-tls\") pod \"image-registry-566dbff8bb-sf4f6\" (UID: \"d2d9ad62-a00d-4564-b22b-78e932bfb42c\") " pod="openshift-image-registry/image-registry-566dbff8bb-sf4f6" Nov 03 20:39:39.052231 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:39.052144 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d2d9ad62-a00d-4564-b22b-78e932bfb42c-installation-pull-secrets\") pod \"image-registry-566dbff8bb-sf4f6\" (UID: \"d2d9ad62-a00d-4564-b22b-78e932bfb42c\") " pod="openshift-image-registry/image-registry-566dbff8bb-sf4f6" Nov 03 20:39:39.052483 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:39.052461 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d2d9ad62-a00d-4564-b22b-78e932bfb42c-image-registry-private-configuration\") pod \"image-registry-566dbff8bb-sf4f6\" (UID: \"d2d9ad62-a00d-4564-b22b-78e932bfb42c\") " pod="openshift-image-registry/image-registry-566dbff8bb-sf4f6" Nov 03 20:39:39.052551 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:39.052504 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/10e7de28-0482-4218-85f6-0960071ee6f4-cert\") pod \"ingress-canary-wfv5k\" (UID: \"10e7de28-0482-4218-85f6-0960071ee6f4\") " pod="openshift-ingress-canary/ingress-canary-wfv5k" Nov 03 20:39:39.085089 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:39.085059 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-b4gn6" Nov 03 20:39:39.114423 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:39.114385 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnlhc\" (UniqueName: \"kubernetes.io/projected/d2d9ad62-a00d-4564-b22b-78e932bfb42c-kube-api-access-xnlhc\") pod \"image-registry-566dbff8bb-sf4f6\" (UID: \"d2d9ad62-a00d-4564-b22b-78e932bfb42c\") " pod="openshift-image-registry/image-registry-566dbff8bb-sf4f6" Nov 03 20:39:39.114592 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:39.114518 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwlhv\" (UniqueName: \"kubernetes.io/projected/10e7de28-0482-4218-85f6-0960071ee6f4-kube-api-access-fwlhv\") pod \"ingress-canary-wfv5k\" (UID: \"10e7de28-0482-4218-85f6-0960071ee6f4\") " pod="openshift-ingress-canary/ingress-canary-wfv5k" Nov 03 20:39:39.115642 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:39.115623 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d2d9ad62-a00d-4564-b22b-78e932bfb42c-bound-sa-token\") pod \"image-registry-566dbff8bb-sf4f6\" (UID: \"d2d9ad62-a00d-4564-b22b-78e932bfb42c\") " pod="openshift-image-registry/image-registry-566dbff8bb-sf4f6" Nov 03 20:39:39.123510 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:39.123479 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9k4g2" Nov 03 20:39:39.228100 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:39.228066 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wfv5k" Nov 03 20:39:39.233666 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:39.233137 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-566dbff8bb-sf4f6" Nov 03 20:39:39.493699 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:39.493668 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-b4gn6"] Nov 03 20:39:39.496580 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:39.496555 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffc31124_d81f_40af_876c_da6c6979714b.slice/crio-7328906cb2be43cc5719feff37b0d93d795969ab080184fb21e5ad24b88fe669 WatchSource:0}: Error finding container 7328906cb2be43cc5719feff37b0d93d795969ab080184fb21e5ad24b88fe669: Status 404 returned error can't find the container with id 7328906cb2be43cc5719feff37b0d93d795969ab080184fb21e5ad24b88fe669 Nov 03 20:39:39.505021 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:39.504947 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9k4g2"] Nov 03 20:39:39.509186 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:39.509154 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod750c5782_16b3_4ab6_a88d_cd2e326fcc07.slice/crio-975ddf26993b580b65c157337cc16146be84ec7a545cd4c983751864bbfdcf49 WatchSource:0}: Error finding container 975ddf26993b580b65c157337cc16146be84ec7a545cd4c983751864bbfdcf49: Status 404 returned error can't find the container with id 975ddf26993b580b65c157337cc16146be84ec7a545cd4c983751864bbfdcf49 Nov 03 20:39:39.606361 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:39.606323 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wfv5k"] Nov 03 20:39:39.610224 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:39.610198 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-566dbff8bb-sf4f6"] Nov 03 20:39:39.610830 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:39.610806 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10e7de28_0482_4218_85f6_0960071ee6f4.slice/crio-d3a517ba918f6fa55ef0e8d12a5b6d02429aab4c0f47e97d067140b979458701 WatchSource:0}: Error finding container d3a517ba918f6fa55ef0e8d12a5b6d02429aab4c0f47e97d067140b979458701: Status 404 returned error can't find the container with id d3a517ba918f6fa55ef0e8d12a5b6d02429aab4c0f47e97d067140b979458701 Nov 03 20:39:39.613621 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:39.613539 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2d9ad62_a00d_4564_b22b_78e932bfb42c.slice/crio-31754082f77d3b9229ac07a23e58679ea5fa02e28b27605396e408598323e398 WatchSource:0}: Error finding container 31754082f77d3b9229ac07a23e58679ea5fa02e28b27605396e408598323e398: Status 404 returned error can't find the container with id 31754082f77d3b9229ac07a23e58679ea5fa02e28b27605396e408598323e398 Nov 03 20:39:39.958996 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:39.958958 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wfv5k" event={"ID":"10e7de28-0482-4218-85f6-0960071ee6f4","Type":"ContainerStarted","Data":"d3a517ba918f6fa55ef0e8d12a5b6d02429aab4c0f47e97d067140b979458701"} Nov 03 20:39:39.960217 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:39.960193 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9k4g2" event={"ID":"750c5782-16b3-4ab6-a88d-cd2e326fcc07","Type":"ContainerStarted","Data":"975ddf26993b580b65c157337cc16146be84ec7a545cd4c983751864bbfdcf49"} Nov 03 20:39:39.961915 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:39.961890 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b4gn6" event={"ID":"ffc31124-d81f-40af-876c-da6c6979714b","Type":"ContainerStarted","Data":"315e7c6c82be722e84fac8edee223349e75e922791b5b47c393ec7742601bb40"} Nov 03 20:39:39.962009 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:39.961951 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b4gn6" event={"ID":"ffc31124-d81f-40af-876c-da6c6979714b","Type":"ContainerStarted","Data":"7328906cb2be43cc5719feff37b0d93d795969ab080184fb21e5ad24b88fe669"} Nov 03 20:39:39.963807 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:39.963783 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-566dbff8bb-sf4f6" event={"ID":"d2d9ad62-a00d-4564-b22b-78e932bfb42c","Type":"ContainerStarted","Data":"5e753ee580ad87e0f404fb9108caa56357c668d0bf8edd53f64368ddfaabe526"} Nov 03 20:39:39.963913 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:39.963820 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-566dbff8bb-sf4f6" event={"ID":"d2d9ad62-a00d-4564-b22b-78e932bfb42c","Type":"ContainerStarted","Data":"31754082f77d3b9229ac07a23e58679ea5fa02e28b27605396e408598323e398"} Nov 03 20:39:39.963966 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:39.963929 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-566dbff8bb-sf4f6" Nov 03 20:39:40.084076 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:40.083884 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-566dbff8bb-sf4f6" podStartSLOduration=2.083865041 podStartE2EDuration="2.083865041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:39:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:39:40.083668387 +0000 UTC m=+15.308988180" watchObservedRunningTime="2026-11-03 20:39:40.083865041 +0000 UTC m=+15.309184832" Nov 03 20:39:40.901828 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:40.901800 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4ljxs" Nov 03 20:39:40.970328 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:40.970236 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b4gn6" event={"ID":"ffc31124-d81f-40af-876c-da6c6979714b","Type":"ContainerStarted","Data":"c94bd3f8ec2711587ab7d5ab68d5b5354de7178594251e1ea0eda62422fab42d"} Nov 03 20:39:41.980116 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:41.980077 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9k4g2" event={"ID":"750c5782-16b3-4ab6-a88d-cd2e326fcc07","Type":"ContainerStarted","Data":"1cc5a0e852689f798a433ca0a11985a129b7cad73b027578553ea4883809ff67"} Nov 03 20:39:42.984623 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:42.984582 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wfv5k" event={"ID":"10e7de28-0482-4218-85f6-0960071ee6f4","Type":"ContainerStarted","Data":"1ca32fa415ab394905ad276aea33c0988352ecfd4bbfe15ed6e2af592d6cb899"} Nov 03 20:39:42.986400 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:42.986373 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9k4g2" event={"ID":"750c5782-16b3-4ab6-a88d-cd2e326fcc07","Type":"ContainerStarted","Data":"2aef65b7e8142188f0169ea372055932b026d8d75873e919df590d68df1e60c9"} Nov 03 20:39:42.986549 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:42.986420 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9k4g2" Nov 03 20:39:43.099352 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:43.099254 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-wfv5k" podStartSLOduration=1.909344264 podStartE2EDuration="5.099239914s" totalImagesPullingTime="3.18989565s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:39:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:39:43.098780075 +0000 UTC m=+18.324099880" watchObservedRunningTime="2026-11-03 20:39:43.099239914 +0000 UTC m=+18.324559697" Nov 03 20:39:43.217550 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:43.217482 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-9k4g2" podStartSLOduration=2.8913860959999997 podStartE2EDuration="5.217461659s" totalImagesPullingTime="2.326075563s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:39:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:39:43.217002141 +0000 UTC m=+18.442321943" watchObservedRunningTime="2026-11-03 20:39:43.217461659 +0000 UTC m=+18.442781445" Nov 03 20:39:44.994818 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:44.994784 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b4gn6" event={"ID":"ffc31124-d81f-40af-876c-da6c6979714b","Type":"ContainerStarted","Data":"521b8a1bbc20876da93849cfb6936c3996c2d89edb53ad0bf043445ace6d72fe"} Nov 03 20:39:45.113598 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:45.113549 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-b4gn6" podStartSLOduration=2.225774294 podStartE2EDuration="7.113535879s" totalImagesPullingTime="4.887761585s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:39:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:39:45.112832607 +0000 UTC m=+20.338152408" watchObservedRunningTime="2026-11-03 20:39:45.113535879 +0000 UTC m=+20.338855669" Nov 03 20:39:46.320289 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:39:46.320254 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 20:39:51.062342 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:51.062308 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-66c6bd48db-bjx28"] Nov 03 20:39:51.064754 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:51.064738 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66c6bd48db-bjx28" Nov 03 20:39:51.117217 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:51.117184 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 03 20:39:51.129558 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:51.129525 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/33077f68-6503-41d4-a597-4f2b403b9340-tmp\") pod \"console-66c6bd48db-bjx28\" (UID: \"33077f68-6503-41d4-a597-4f2b403b9340\") " pod="openshift-console/console-66c6bd48db-bjx28" Nov 03 20:39:51.129727 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:51.129577 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2rqq\" (UniqueName: \"kubernetes.io/projected/33077f68-6503-41d4-a597-4f2b403b9340-kube-api-access-g2rqq\") pod \"console-66c6bd48db-bjx28\" (UID: \"33077f68-6503-41d4-a597-4f2b403b9340\") " pod="openshift-console/console-66c6bd48db-bjx28" Nov 03 20:39:51.129727 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:51.129667 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/33077f68-6503-41d4-a597-4f2b403b9340-console-serving-cert\") pod \"console-66c6bd48db-bjx28\" (UID: \"33077f68-6503-41d4-a597-4f2b403b9340\") " pod="openshift-console/console-66c6bd48db-bjx28" Nov 03 20:39:51.129854 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:51.129727 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/33077f68-6503-41d4-a597-4f2b403b9340-oauth-serving-cert\") pod \"console-66c6bd48db-bjx28\" (UID: \"33077f68-6503-41d4-a597-4f2b403b9340\") " pod="openshift-console/console-66c6bd48db-bjx28" Nov 03 20:39:51.129854 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:51.129758 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/33077f68-6503-41d4-a597-4f2b403b9340-console-config\") pod \"console-66c6bd48db-bjx28\" (UID: \"33077f68-6503-41d4-a597-4f2b403b9340\") " pod="openshift-console/console-66c6bd48db-bjx28" Nov 03 20:39:51.129854 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:51.129836 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/33077f68-6503-41d4-a597-4f2b403b9340-service-ca\") pod \"console-66c6bd48db-bjx28\" (UID: \"33077f68-6503-41d4-a597-4f2b403b9340\") " pod="openshift-console/console-66c6bd48db-bjx28" Nov 03 20:39:51.129971 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:51.129868 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/33077f68-6503-41d4-a597-4f2b403b9340-console-oauth-config\") pod \"console-66c6bd48db-bjx28\" (UID: \"33077f68-6503-41d4-a597-4f2b403b9340\") " pod="openshift-console/console-66c6bd48db-bjx28" Nov 03 20:39:51.166782 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:51.166748 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 03 20:39:51.166782 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:51.166752 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 03 20:39:51.166782 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:51.166803 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 03 20:39:51.167034 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:51.166845 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 03 20:39:51.167034 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:51.166840 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 03 20:39:51.167034 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:51.166869 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 03 20:39:51.167120 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:51.167037 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-ltwhn\"" Nov 03 20:39:51.167745 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:51.167729 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66c6bd48db-bjx28"] Nov 03 20:39:51.230802 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:51.230765 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/33077f68-6503-41d4-a597-4f2b403b9340-console-serving-cert\") pod \"console-66c6bd48db-bjx28\" (UID: \"33077f68-6503-41d4-a597-4f2b403b9340\") " pod="openshift-console/console-66c6bd48db-bjx28" Nov 03 20:39:51.230802 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:51.230796 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/33077f68-6503-41d4-a597-4f2b403b9340-oauth-serving-cert\") pod \"console-66c6bd48db-bjx28\" (UID: \"33077f68-6503-41d4-a597-4f2b403b9340\") " pod="openshift-console/console-66c6bd48db-bjx28" Nov 03 20:39:51.231017 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:51.230817 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/33077f68-6503-41d4-a597-4f2b403b9340-console-config\") pod \"console-66c6bd48db-bjx28\" (UID: \"33077f68-6503-41d4-a597-4f2b403b9340\") " pod="openshift-console/console-66c6bd48db-bjx28" Nov 03 20:39:51.231017 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:51.230839 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/33077f68-6503-41d4-a597-4f2b403b9340-service-ca\") pod \"console-66c6bd48db-bjx28\" (UID: \"33077f68-6503-41d4-a597-4f2b403b9340\") " pod="openshift-console/console-66c6bd48db-bjx28" Nov 03 20:39:51.231017 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:51.230974 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/33077f68-6503-41d4-a597-4f2b403b9340-console-oauth-config\") pod \"console-66c6bd48db-bjx28\" (UID: \"33077f68-6503-41d4-a597-4f2b403b9340\") " pod="openshift-console/console-66c6bd48db-bjx28" Nov 03 20:39:51.231166 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:51.231031 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/33077f68-6503-41d4-a597-4f2b403b9340-tmp\") pod \"console-66c6bd48db-bjx28\" (UID: \"33077f68-6503-41d4-a597-4f2b403b9340\") " pod="openshift-console/console-66c6bd48db-bjx28" Nov 03 20:39:51.231166 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:51.231063 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2rqq\" (UniqueName: \"kubernetes.io/projected/33077f68-6503-41d4-a597-4f2b403b9340-kube-api-access-g2rqq\") pod \"console-66c6bd48db-bjx28\" (UID: \"33077f68-6503-41d4-a597-4f2b403b9340\") " pod="openshift-console/console-66c6bd48db-bjx28" Nov 03 20:39:51.231454 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:51.231407 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/33077f68-6503-41d4-a597-4f2b403b9340-tmp\") pod \"console-66c6bd48db-bjx28\" (UID: \"33077f68-6503-41d4-a597-4f2b403b9340\") " pod="openshift-console/console-66c6bd48db-bjx28" Nov 03 20:39:51.231694 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:51.231676 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/33077f68-6503-41d4-a597-4f2b403b9340-oauth-serving-cert\") pod \"console-66c6bd48db-bjx28\" (UID: \"33077f68-6503-41d4-a597-4f2b403b9340\") " pod="openshift-console/console-66c6bd48db-bjx28" Nov 03 20:39:51.231780 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:51.231723 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/33077f68-6503-41d4-a597-4f2b403b9340-service-ca\") pod \"console-66c6bd48db-bjx28\" (UID: \"33077f68-6503-41d4-a597-4f2b403b9340\") " pod="openshift-console/console-66c6bd48db-bjx28" Nov 03 20:39:51.231891 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:51.231865 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/33077f68-6503-41d4-a597-4f2b403b9340-console-config\") pod \"console-66c6bd48db-bjx28\" (UID: \"33077f68-6503-41d4-a597-4f2b403b9340\") " pod="openshift-console/console-66c6bd48db-bjx28" Nov 03 20:39:51.233478 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:51.233460 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/33077f68-6503-41d4-a597-4f2b403b9340-console-serving-cert\") pod \"console-66c6bd48db-bjx28\" (UID: \"33077f68-6503-41d4-a597-4f2b403b9340\") " pod="openshift-console/console-66c6bd48db-bjx28" Nov 03 20:39:51.233571 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:51.233489 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/33077f68-6503-41d4-a597-4f2b403b9340-console-oauth-config\") pod \"console-66c6bd48db-bjx28\" (UID: \"33077f68-6503-41d4-a597-4f2b403b9340\") " pod="openshift-console/console-66c6bd48db-bjx28" Nov 03 20:39:51.292179 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:51.292150 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2rqq\" (UniqueName: \"kubernetes.io/projected/33077f68-6503-41d4-a597-4f2b403b9340-kube-api-access-g2rqq\") pod \"console-66c6bd48db-bjx28\" (UID: \"33077f68-6503-41d4-a597-4f2b403b9340\") " pod="openshift-console/console-66c6bd48db-bjx28" Nov 03 20:39:51.374480 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:51.374369 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66c6bd48db-bjx28" Nov 03 20:39:51.748629 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:51.748597 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66c6bd48db-bjx28"] Nov 03 20:39:51.751259 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:51.751228 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33077f68_6503_41d4_a597_4f2b403b9340.slice/crio-110a70d3cc183bad7461152cc326f46fa69683de22ed2e80cd728eec5209f272 WatchSource:0}: Error finding container 110a70d3cc183bad7461152cc326f46fa69683de22ed2e80cd728eec5209f272: Status 404 returned error can't find the container with id 110a70d3cc183bad7461152cc326f46fa69683de22ed2e80cd728eec5209f272 Nov 03 20:39:52.020974 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:52.020892 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66c6bd48db-bjx28" event={"ID":"33077f68-6503-41d4-a597-4f2b403b9340","Type":"ContainerStarted","Data":"110a70d3cc183bad7461152cc326f46fa69683de22ed2e80cd728eec5209f272"} Nov 03 20:39:52.993185 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:52.993157 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9k4g2" Nov 03 20:39:55.148958 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.148921 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-xsn6d"] Nov 03 20:39:55.152093 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.151225 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xsn6d" Nov 03 20:39:55.198862 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.198824 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-tv9dk"] Nov 03 20:39:55.201567 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.201538 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tv9dk" Nov 03 20:39:55.247460 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.247382 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 03 20:39:55.247460 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.247383 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 03 20:39:55.247460 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.247422 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 20:39:55.259722 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.259690 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8f54b118-294c-48cf-9bd7-20a2602043d2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-tv9dk\" (UID: \"8f54b118-294c-48cf-9bd7-20a2602043d2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tv9dk" Nov 03 20:39:55.259893 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.259737 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f54b118-294c-48cf-9bd7-20a2602043d2-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-tv9dk\" (UID: \"8f54b118-294c-48cf-9bd7-20a2602043d2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tv9dk" Nov 03 20:39:55.259893 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.259766 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8f54b118-294c-48cf-9bd7-20a2602043d2-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-tv9dk\" (UID: \"8f54b118-294c-48cf-9bd7-20a2602043d2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tv9dk" Nov 03 20:39:55.259999 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.259910 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/592c07dc-3fef-4f51-87a4-669769c5a9ef-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-xsn6d\" (UID: \"592c07dc-3fef-4f51-87a4-669769c5a9ef\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xsn6d" Nov 03 20:39:55.259999 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.259950 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/592c07dc-3fef-4f51-87a4-669769c5a9ef-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-xsn6d\" (UID: \"592c07dc-3fef-4f51-87a4-669769c5a9ef\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xsn6d" Nov 03 20:39:55.259999 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.259978 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg254\" (UniqueName: \"kubernetes.io/projected/8f54b118-294c-48cf-9bd7-20a2602043d2-kube-api-access-zg254\") pod \"kube-state-metrics-5dc8d7d89b-tv9dk\" (UID: \"8f54b118-294c-48cf-9bd7-20a2602043d2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tv9dk" Nov 03 20:39:55.260127 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.260039 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4d5j\" (UniqueName: \"kubernetes.io/projected/592c07dc-3fef-4f51-87a4-669769c5a9ef-kube-api-access-q4d5j\") pod \"openshift-state-metrics-7bd7489957-xsn6d\" (UID: \"592c07dc-3fef-4f51-87a4-669769c5a9ef\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xsn6d" Nov 03 20:39:55.260127 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.260062 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8f54b118-294c-48cf-9bd7-20a2602043d2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-tv9dk\" (UID: \"8f54b118-294c-48cf-9bd7-20a2602043d2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tv9dk" Nov 03 20:39:55.260226 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.260170 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/592c07dc-3fef-4f51-87a4-669769c5a9ef-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-xsn6d\" (UID: \"592c07dc-3fef-4f51-87a4-669769c5a9ef\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xsn6d" Nov 03 20:39:55.260226 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.260198 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f54b118-294c-48cf-9bd7-20a2602043d2-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-tv9dk\" (UID: \"8f54b118-294c-48cf-9bd7-20a2602043d2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tv9dk" Nov 03 20:39:55.262200 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.262179 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 20:39:55.262331 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.262255 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 20:39:55.262331 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.262266 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-rz99s\"" Nov 03 20:39:55.263059 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.263040 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2s79k"] Nov 03 20:39:55.265607 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.265579 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2s79k" Nov 03 20:39:55.294348 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.294311 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-xsn6d"] Nov 03 20:39:55.294963 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.294943 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-tv9dk"] Nov 03 20:39:55.313149 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.313099 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 03 20:39:55.313534 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.313506 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 03 20:39:55.313534 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.313511 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 03 20:39:55.313688 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.313549 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-8r7ss\"" Nov 03 20:39:55.350465 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.350416 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 20:39:55.360725 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.360698 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4d5j\" (UniqueName: \"kubernetes.io/projected/592c07dc-3fef-4f51-87a4-669769c5a9ef-kube-api-access-q4d5j\") pod \"openshift-state-metrics-7bd7489957-xsn6d\" (UID: \"592c07dc-3fef-4f51-87a4-669769c5a9ef\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xsn6d" Nov 03 20:39:55.360886 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.360740 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8f54b118-294c-48cf-9bd7-20a2602043d2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-tv9dk\" (UID: \"8f54b118-294c-48cf-9bd7-20a2602043d2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tv9dk" Nov 03 20:39:55.360886 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.360765 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac7e0d69-b7d1-4b23-9526-71e488829501-sys\") pod \"node-exporter-2s79k\" (UID: \"ac7e0d69-b7d1-4b23-9526-71e488829501\") " pod="openshift-monitoring/node-exporter-2s79k" Nov 03 20:39:55.360886 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.360802 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/592c07dc-3fef-4f51-87a4-669769c5a9ef-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-xsn6d\" (UID: \"592c07dc-3fef-4f51-87a4-669769c5a9ef\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xsn6d" Nov 03 20:39:55.360886 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.360828 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac7e0d69-b7d1-4b23-9526-71e488829501-metrics-client-ca\") pod \"node-exporter-2s79k\" (UID: \"ac7e0d69-b7d1-4b23-9526-71e488829501\") " pod="openshift-monitoring/node-exporter-2s79k" Nov 03 20:39:55.360886 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.360847 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f54b118-294c-48cf-9bd7-20a2602043d2-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-tv9dk\" (UID: \"8f54b118-294c-48cf-9bd7-20a2602043d2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tv9dk" Nov 03 20:39:55.360886 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.360873 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mckt\" (UniqueName: \"kubernetes.io/projected/ac7e0d69-b7d1-4b23-9526-71e488829501-kube-api-access-6mckt\") pod \"node-exporter-2s79k\" (UID: \"ac7e0d69-b7d1-4b23-9526-71e488829501\") " pod="openshift-monitoring/node-exporter-2s79k" Nov 03 20:39:55.361179 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.360905 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ac7e0d69-b7d1-4b23-9526-71e488829501-node-exporter-textfile\") pod \"node-exporter-2s79k\" (UID: \"ac7e0d69-b7d1-4b23-9526-71e488829501\") " pod="openshift-monitoring/node-exporter-2s79k" Nov 03 20:39:55.361179 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.360936 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8f54b118-294c-48cf-9bd7-20a2602043d2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-tv9dk\" (UID: \"8f54b118-294c-48cf-9bd7-20a2602043d2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tv9dk" Nov 03 20:39:55.361179 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.360967 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ac7e0d69-b7d1-4b23-9526-71e488829501-node-exporter-accelerators-collector-config\") pod \"node-exporter-2s79k\" (UID: \"ac7e0d69-b7d1-4b23-9526-71e488829501\") " pod="openshift-monitoring/node-exporter-2s79k" Nov 03 20:39:55.361179 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.361038 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ac7e0d69-b7d1-4b23-9526-71e488829501-node-exporter-wtmp\") pod \"node-exporter-2s79k\" (UID: \"ac7e0d69-b7d1-4b23-9526-71e488829501\") " pod="openshift-monitoring/node-exporter-2s79k" Nov 03 20:39:55.361179 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.361119 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f54b118-294c-48cf-9bd7-20a2602043d2-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-tv9dk\" (UID: \"8f54b118-294c-48cf-9bd7-20a2602043d2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tv9dk" Nov 03 20:39:55.361420 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.361236 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ac7e0d69-b7d1-4b23-9526-71e488829501-node-exporter-tls\") pod \"node-exporter-2s79k\" (UID: \"ac7e0d69-b7d1-4b23-9526-71e488829501\") " pod="openshift-monitoring/node-exporter-2s79k" Nov 03 20:39:55.361420 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.361312 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8f54b118-294c-48cf-9bd7-20a2602043d2-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-tv9dk\" (UID: \"8f54b118-294c-48cf-9bd7-20a2602043d2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tv9dk" Nov 03 20:39:55.361567 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.361417 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8f54b118-294c-48cf-9bd7-20a2602043d2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-tv9dk\" (UID: \"8f54b118-294c-48cf-9bd7-20a2602043d2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tv9dk" Nov 03 20:39:55.361567 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.361508 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/592c07dc-3fef-4f51-87a4-669769c5a9ef-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-xsn6d\" (UID: \"592c07dc-3fef-4f51-87a4-669769c5a9ef\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xsn6d" Nov 03 20:39:55.361663 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.361565 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ac7e0d69-b7d1-4b23-9526-71e488829501-root\") pod \"node-exporter-2s79k\" (UID: \"ac7e0d69-b7d1-4b23-9526-71e488829501\") " pod="openshift-monitoring/node-exporter-2s79k" Nov 03 20:39:55.361663 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.361628 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ac7e0d69-b7d1-4b23-9526-71e488829501-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2s79k\" (UID: \"ac7e0d69-b7d1-4b23-9526-71e488829501\") " pod="openshift-monitoring/node-exporter-2s79k" Nov 03 20:39:55.361761 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.361696 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/592c07dc-3fef-4f51-87a4-669769c5a9ef-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-xsn6d\" (UID: \"592c07dc-3fef-4f51-87a4-669769c5a9ef\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xsn6d" Nov 03 20:39:55.361761 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.361722 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg254\" (UniqueName: \"kubernetes.io/projected/8f54b118-294c-48cf-9bd7-20a2602043d2-kube-api-access-zg254\") pod \"kube-state-metrics-5dc8d7d89b-tv9dk\" (UID: \"8f54b118-294c-48cf-9bd7-20a2602043d2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tv9dk" Nov 03 20:39:55.361923 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.361904 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8f54b118-294c-48cf-9bd7-20a2602043d2-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-tv9dk\" (UID: \"8f54b118-294c-48cf-9bd7-20a2602043d2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tv9dk" Nov 03 20:39:55.361987 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.361952 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f54b118-294c-48cf-9bd7-20a2602043d2-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-tv9dk\" (UID: \"8f54b118-294c-48cf-9bd7-20a2602043d2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tv9dk" Nov 03 20:39:55.362584 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.362507 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/592c07dc-3fef-4f51-87a4-669769c5a9ef-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-xsn6d\" (UID: \"592c07dc-3fef-4f51-87a4-669769c5a9ef\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xsn6d" Nov 03 20:39:55.364209 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.364188 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/592c07dc-3fef-4f51-87a4-669769c5a9ef-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-xsn6d\" (UID: \"592c07dc-3fef-4f51-87a4-669769c5a9ef\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xsn6d" Nov 03 20:39:55.364315 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.364215 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f54b118-294c-48cf-9bd7-20a2602043d2-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-tv9dk\" (UID: \"8f54b118-294c-48cf-9bd7-20a2602043d2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tv9dk" Nov 03 20:39:55.364401 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.364378 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/592c07dc-3fef-4f51-87a4-669769c5a9ef-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-xsn6d\" (UID: \"592c07dc-3fef-4f51-87a4-669769c5a9ef\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xsn6d" Nov 03 20:39:55.364678 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.364659 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8f54b118-294c-48cf-9bd7-20a2602043d2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-tv9dk\" (UID: \"8f54b118-294c-48cf-9bd7-20a2602043d2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tv9dk" Nov 03 20:39:55.413805 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.413730 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 20:39:55.413970 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.413824 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rzwv7\"" Nov 03 20:39:55.413970 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.413844 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 20:39:55.423930 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.423906 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4d5j\" (UniqueName: \"kubernetes.io/projected/592c07dc-3fef-4f51-87a4-669769c5a9ef-kube-api-access-q4d5j\") pod \"openshift-state-metrics-7bd7489957-xsn6d\" (UID: \"592c07dc-3fef-4f51-87a4-669769c5a9ef\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xsn6d" Nov 03 20:39:55.423930 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.423926 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg254\" (UniqueName: \"kubernetes.io/projected/8f54b118-294c-48cf-9bd7-20a2602043d2-kube-api-access-zg254\") pod \"kube-state-metrics-5dc8d7d89b-tv9dk\" (UID: \"8f54b118-294c-48cf-9bd7-20a2602043d2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tv9dk" Nov 03 20:39:55.462459 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.462408 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac7e0d69-b7d1-4b23-9526-71e488829501-metrics-client-ca\") pod \"node-exporter-2s79k\" (UID: \"ac7e0d69-b7d1-4b23-9526-71e488829501\") " pod="openshift-monitoring/node-exporter-2s79k" Nov 03 20:39:55.462627 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.462484 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mckt\" (UniqueName: \"kubernetes.io/projected/ac7e0d69-b7d1-4b23-9526-71e488829501-kube-api-access-6mckt\") pod \"node-exporter-2s79k\" (UID: \"ac7e0d69-b7d1-4b23-9526-71e488829501\") " pod="openshift-monitoring/node-exporter-2s79k" Nov 03 20:39:55.462627 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.462517 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ac7e0d69-b7d1-4b23-9526-71e488829501-node-exporter-textfile\") pod \"node-exporter-2s79k\" (UID: \"ac7e0d69-b7d1-4b23-9526-71e488829501\") " pod="openshift-monitoring/node-exporter-2s79k" Nov 03 20:39:55.462627 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.462553 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ac7e0d69-b7d1-4b23-9526-71e488829501-node-exporter-accelerators-collector-config\") pod \"node-exporter-2s79k\" (UID: \"ac7e0d69-b7d1-4b23-9526-71e488829501\") " pod="openshift-monitoring/node-exporter-2s79k" Nov 03 20:39:55.462627 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.462588 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ac7e0d69-b7d1-4b23-9526-71e488829501-node-exporter-wtmp\") pod \"node-exporter-2s79k\" (UID: \"ac7e0d69-b7d1-4b23-9526-71e488829501\") " pod="openshift-monitoring/node-exporter-2s79k" Nov 03 20:39:55.462627 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.462610 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ac7e0d69-b7d1-4b23-9526-71e488829501-node-exporter-tls\") pod \"node-exporter-2s79k\" (UID: \"ac7e0d69-b7d1-4b23-9526-71e488829501\") " pod="openshift-monitoring/node-exporter-2s79k" Nov 03 20:39:55.462861 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.462646 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ac7e0d69-b7d1-4b23-9526-71e488829501-root\") pod \"node-exporter-2s79k\" (UID: \"ac7e0d69-b7d1-4b23-9526-71e488829501\") " pod="openshift-monitoring/node-exporter-2s79k" Nov 03 20:39:55.462861 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.462673 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ac7e0d69-b7d1-4b23-9526-71e488829501-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2s79k\" (UID: \"ac7e0d69-b7d1-4b23-9526-71e488829501\") " pod="openshift-monitoring/node-exporter-2s79k" Nov 03 20:39:55.462861 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.462763 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac7e0d69-b7d1-4b23-9526-71e488829501-sys\") pod \"node-exporter-2s79k\" (UID: \"ac7e0d69-b7d1-4b23-9526-71e488829501\") " pod="openshift-monitoring/node-exporter-2s79k" Nov 03 20:39:55.462861 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.462847 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac7e0d69-b7d1-4b23-9526-71e488829501-sys\") pod \"node-exporter-2s79k\" (UID: \"ac7e0d69-b7d1-4b23-9526-71e488829501\") " pod="openshift-monitoring/node-exporter-2s79k" Nov 03 20:39:55.463086 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.463065 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac7e0d69-b7d1-4b23-9526-71e488829501-metrics-client-ca\") pod \"node-exporter-2s79k\" (UID: \"ac7e0d69-b7d1-4b23-9526-71e488829501\") " pod="openshift-monitoring/node-exporter-2s79k" Nov 03 20:39:55.463144 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.463098 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ac7e0d69-b7d1-4b23-9526-71e488829501-node-exporter-textfile\") pod \"node-exporter-2s79k\" (UID: \"ac7e0d69-b7d1-4b23-9526-71e488829501\") " pod="openshift-monitoring/node-exporter-2s79k" Nov 03 20:39:55.463144 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.463132 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ac7e0d69-b7d1-4b23-9526-71e488829501-root\") pod \"node-exporter-2s79k\" (UID: \"ac7e0d69-b7d1-4b23-9526-71e488829501\") " pod="openshift-monitoring/node-exporter-2s79k" Nov 03 20:39:55.463384 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.463357 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ac7e0d69-b7d1-4b23-9526-71e488829501-node-exporter-wtmp\") pod \"node-exporter-2s79k\" (UID: \"ac7e0d69-b7d1-4b23-9526-71e488829501\") " pod="openshift-monitoring/node-exporter-2s79k" Nov 03 20:39:55.463793 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.463597 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ac7e0d69-b7d1-4b23-9526-71e488829501-node-exporter-accelerators-collector-config\") pod \"node-exporter-2s79k\" (UID: \"ac7e0d69-b7d1-4b23-9526-71e488829501\") " pod="openshift-monitoring/node-exporter-2s79k" Nov 03 20:39:55.465517 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.465494 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xsn6d" Nov 03 20:39:55.465721 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.465701 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ac7e0d69-b7d1-4b23-9526-71e488829501-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2s79k\" (UID: \"ac7e0d69-b7d1-4b23-9526-71e488829501\") " pod="openshift-monitoring/node-exporter-2s79k" Nov 03 20:39:55.465895 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.465879 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ac7e0d69-b7d1-4b23-9526-71e488829501-node-exporter-tls\") pod \"node-exporter-2s79k\" (UID: \"ac7e0d69-b7d1-4b23-9526-71e488829501\") " pod="openshift-monitoring/node-exporter-2s79k" Nov 03 20:39:55.516892 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.516855 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tv9dk" Nov 03 20:39:55.529639 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.524474 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mckt\" (UniqueName: \"kubernetes.io/projected/ac7e0d69-b7d1-4b23-9526-71e488829501-kube-api-access-6mckt\") pod \"node-exporter-2s79k\" (UID: \"ac7e0d69-b7d1-4b23-9526-71e488829501\") " pod="openshift-monitoring/node-exporter-2s79k" Nov 03 20:39:55.580945 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.580921 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2s79k" Nov 03 20:39:55.890734 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.890656 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-xsn6d"] Nov 03 20:39:55.895229 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:55.895193 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod592c07dc_3fef_4f51_87a4_669769c5a9ef.slice/crio-dc8e0a47baa34e18f9dd40fc6413aae168a0bec87915b3f5c185a47d89b52454 WatchSource:0}: Error finding container dc8e0a47baa34e18f9dd40fc6413aae168a0bec87915b3f5c185a47d89b52454: Status 404 returned error can't find the container with id dc8e0a47baa34e18f9dd40fc6413aae168a0bec87915b3f5c185a47d89b52454 Nov 03 20:39:55.944715 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:55.944685 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-tv9dk"] Nov 03 20:39:55.948666 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:55.948637 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f54b118_294c_48cf_9bd7_20a2602043d2.slice/crio-6f952da595c2ff33fb05083e3e05fef774d88fb1ac7c7f5eac01410411c704e3 WatchSource:0}: Error finding container 6f952da595c2ff33fb05083e3e05fef774d88fb1ac7c7f5eac01410411c704e3: Status 404 returned error can't find the container with id 6f952da595c2ff33fb05083e3e05fef774d88fb1ac7c7f5eac01410411c704e3 Nov 03 20:39:56.036445 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:56.036392 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tv9dk" event={"ID":"8f54b118-294c-48cf-9bd7-20a2602043d2","Type":"ContainerStarted","Data":"6f952da595c2ff33fb05083e3e05fef774d88fb1ac7c7f5eac01410411c704e3"} Nov 03 20:39:56.038608 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:56.038579 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xsn6d" event={"ID":"592c07dc-3fef-4f51-87a4-669769c5a9ef","Type":"ContainerStarted","Data":"81d4b3f4a1538d821d69c40635d763d01bf45266249d7af21ee8250ae938245e"} Nov 03 20:39:56.038747 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:56.038636 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xsn6d" event={"ID":"592c07dc-3fef-4f51-87a4-669769c5a9ef","Type":"ContainerStarted","Data":"dc8e0a47baa34e18f9dd40fc6413aae168a0bec87915b3f5c185a47d89b52454"} Nov 03 20:39:56.040133 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:56.040104 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2s79k" event={"ID":"ac7e0d69-b7d1-4b23-9526-71e488829501","Type":"ContainerStarted","Data":"22a7322f2f5d7389bb530c4bad8f7430d7417c76a91933ddbc9a73ba08c260a8"} Nov 03 20:39:57.048045 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:57.047953 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66c6bd48db-bjx28" event={"ID":"33077f68-6503-41d4-a597-4f2b403b9340","Type":"ContainerStarted","Data":"4ddba416c5b5733368960574da821a5e7a935ec132c5a5617163be3fdcf19dcb"} Nov 03 20:39:57.050365 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:57.050332 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xsn6d" event={"ID":"592c07dc-3fef-4f51-87a4-669769c5a9ef","Type":"ContainerStarted","Data":"6711198c69d9ede82e0e85a784cbc3ea385e292fdc8ccb5acc0b5368f9832eea"} Nov 03 20:39:57.405547 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:57.405488 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-66c6bd48db-bjx28" podStartSLOduration=1.466044879 podStartE2EDuration="6.405467727s" totalImagesPullingTime="4.939422848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:39:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:39:57.166875174 +0000 UTC m=+32.392194965" watchObservedRunningTime="2026-11-03 20:39:57.405467727 +0000 UTC m=+32.630787513" Nov 03 20:39:57.406252 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:57.406233 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 20:39:57.418596 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:57.418565 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:39:57.471360 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:57.471324 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 03 20:39:57.471568 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:57.471389 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 03 20:39:57.471648 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:57.471581 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 03 20:39:57.471648 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:57.471602 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 03 20:39:57.471766 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:57.471648 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 03 20:39:57.471766 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:57.471684 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 03 20:39:57.471766 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:57.471697 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-65c9p\"" Nov 03 20:39:57.474787 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:57.474768 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 03 20:39:57.478735 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:57.478714 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 20:39:57.481739 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:57.481708 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:39:57.481867 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:57.481752 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbl8b\" (UniqueName: \"kubernetes.io/projected/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-kube-api-access-xbl8b\") pod \"alertmanager-main-0\" (UID: \"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:39:57.481867 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:57.481782 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:39:57.481867 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:57.481859 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-web-config\") pod \"alertmanager-main-0\" (UID: \"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:39:57.482024 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:57.481901 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:39:57.482024 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:57.481924 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:39:57.482024 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:57.481946 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:39:57.482024 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:57.481999 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-config-out\") pod \"alertmanager-main-0\" (UID: \"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:39:57.482215 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:57.482093 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:39:57.482215 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:57.482121 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:39:57.482215 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:57.482168 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:39:57.482215 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:57.482186 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-config-volume\") pod \"alertmanager-main-0\" (UID: \"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:39:57.482336 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:57.482266 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:39:57.526080 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:57.526047 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 03 20:39:57.571392 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:57.571357 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 03 20:39:57.582667 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:57.582642 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:39:57.582790 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:57.582684 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-config-out\") pod \"alertmanager-main-0\" (UID: \"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:39:57.582790 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:57.582716 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:39:57.582790 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:57.582744 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:39:57.582790 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:57.582772 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:39:57.582993 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:57.582797 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-config-volume\") pod \"alertmanager-main-0\" (UID: \"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:39:57.582993 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:57.582859 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:39:57.582993 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:57.582900 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:39:57.582993 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:57.582926 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbl8b\" (UniqueName: \"kubernetes.io/projected/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-kube-api-access-xbl8b\") pod \"alertmanager-main-0\" (UID: \"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:39:57.583177 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:57.582991 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:39:57.583177 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:57.583022 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-web-config\") pod \"alertmanager-main-0\" (UID: \"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:39:57.583177 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:57.583049 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:39:57.583177 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:57.583075 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:39:57.583375 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:57.583174 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:39:57.584090 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:57.584022 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:39:57.584376 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:57.584350 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:39:57.585784 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:57.585757 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-config-out\") pod \"alertmanager-main-0\" (UID: \"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:39:57.586334 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:57.586225 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:39:57.586972 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:57.586949 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:39:57.587077 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:57.587026 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:39:57.587263 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:57.587224 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:39:57.587979 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:57.587954 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-config-volume\") pod \"alertmanager-main-0\" (UID: \"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:39:57.588084 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:57.587962 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:39:57.588367 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:57.588325 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-web-config\") pod \"alertmanager-main-0\" (UID: \"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:39:57.588494 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:57.588475 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:39:57.650005 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:57.649977 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbl8b\" (UniqueName: \"kubernetes.io/projected/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-kube-api-access-xbl8b\") pod \"alertmanager-main-0\" (UID: \"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:39:57.840139 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:57.839920 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:39:58.055449 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:58.055354 4305 generic.go:356] "Generic (PLEG): container finished" podID="ac7e0d69-b7d1-4b23-9526-71e488829501" containerID="ee22132a36858783cb773f9347e07cee62c2fbc29929fba13bdccb0684e1b425" exitCode=0 Nov 03 20:39:58.055905 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:58.055456 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2s79k" event={"ID":"ac7e0d69-b7d1-4b23-9526-71e488829501","Type":"ContainerDied","Data":"ee22132a36858783cb773f9347e07cee62c2fbc29929fba13bdccb0684e1b425"} Nov 03 20:39:58.347258 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:58.347183 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 20:39:58.453095 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:39:58.453061 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b1935d2_f61f_4c0b_9ba4_e94ebb3575ca.slice/crio-718a4392b03fab4a5ce2fb99b0158cf739c31d16ab67c1b6aad9068c083fa0c2 WatchSource:0}: Error finding container 718a4392b03fab4a5ce2fb99b0158cf739c31d16ab67c1b6aad9068c083fa0c2: Status 404 returned error can't find the container with id 718a4392b03fab4a5ce2fb99b0158cf739c31d16ab67c1b6aad9068c083fa0c2 Nov 03 20:39:59.061022 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:59.060991 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tv9dk" event={"ID":"8f54b118-294c-48cf-9bd7-20a2602043d2","Type":"ContainerStarted","Data":"968f99390846e7f59121d004e4dd19bd3694fc11fc25980d702966d1ca72ce6f"} Nov 03 20:39:59.061509 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:59.061050 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tv9dk" event={"ID":"8f54b118-294c-48cf-9bd7-20a2602043d2","Type":"ContainerStarted","Data":"ccf6882cf6a6417e9ae350ebfe5f8d86e76121e3df6d8426656c1451f43d4350"} Nov 03 20:39:59.061509 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:59.061084 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tv9dk" event={"ID":"8f54b118-294c-48cf-9bd7-20a2602043d2","Type":"ContainerStarted","Data":"ea81a423dc989d88e689624b902c20b5f9cb7d26ab3e0ffd16b9588b5841c6ed"} Nov 03 20:39:59.063196 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:59.063173 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xsn6d" event={"ID":"592c07dc-3fef-4f51-87a4-669769c5a9ef","Type":"ContainerStarted","Data":"7e20fddc29318a83bd6d71fbb28cba27907250326c81c610f9c5df7f9f15130a"} Nov 03 20:39:59.065372 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:59.065340 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2s79k" event={"ID":"ac7e0d69-b7d1-4b23-9526-71e488829501","Type":"ContainerStarted","Data":"7bcb15e32ef98662b669ace7b93225f9ecc40eee8c738cec9e594cd770c0c14f"} Nov 03 20:39:59.065486 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:59.065388 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2s79k" event={"ID":"ac7e0d69-b7d1-4b23-9526-71e488829501","Type":"ContainerStarted","Data":"fd969fd0dd46fe859207a4d96061d7280b6cbbbebf4042cd944d103af82e2803"} Nov 03 20:39:59.066483 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:59.066463 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca","Type":"ContainerStarted","Data":"718a4392b03fab4a5ce2fb99b0158cf739c31d16ab67c1b6aad9068c083fa0c2"} Nov 03 20:39:59.183981 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:59.183872 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-tv9dk" podStartSLOduration=1.461393525 podStartE2EDuration="4.183856378s" totalImagesPullingTime="2.722462853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:39:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:39:59.18129336 +0000 UTC m=+34.406613151" watchObservedRunningTime="2026-11-03 20:39:59.183856378 +0000 UTC m=+34.409176168" Nov 03 20:39:59.237828 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:59.237789 4305 patch_prober.go:29] interesting pod/image-registry-566dbff8bb-sf4f6 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 03 20:39:59.237985 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:59.237839 4305 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-566dbff8bb-sf4f6" podUID="d2d9ad62-a00d-4564-b22b-78e932bfb42c" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:39:59.299717 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:59.299651 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-xsn6d" podStartSLOduration=1.764476353 podStartE2EDuration="4.299636296s" totalImagesPullingTime="2.535159943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:39:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:39:59.299133771 +0000 UTC m=+34.524453581" watchObservedRunningTime="2026-11-03 20:39:59.299636296 +0000 UTC m=+34.524956088" Nov 03 20:39:59.418497 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:59.418417 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2s79k" podStartSLOduration=2.239542369 podStartE2EDuration="4.418399336s" totalImagesPullingTime="2.072757063s" totalInitContainerRuntime="106.099904ms" isStatefulPod=true podCreationTimestamp="2026-11-03 20:39:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:39:59.417167116 +0000 UTC m=+34.642486908" watchObservedRunningTime="2026-11-03 20:39:59.418399336 +0000 UTC m=+34.643719126" Nov 03 20:39:59.813560 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:59.813526 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-hc89p" Nov 03 20:39:59.914525 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:59.914489 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-tjq2x"] Nov 03 20:39:59.936771 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:59.936720 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-tjq2x" Nov 03 20:39:59.975246 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:59.975211 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-tjq2x"] Nov 03 20:39:59.989424 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:59.989395 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-fzvhb\"" Nov 03 20:39:59.989669 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:39:59.989483 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 03 20:40:00.006870 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:00.006835 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/98aa29ea-e74b-41ce-ab99-18b3b7e84338-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-tjq2x\" (UID: \"98aa29ea-e74b-41ce-ab99-18b3b7e84338\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-tjq2x" Nov 03 20:40:00.007030 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:00.006892 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpvzj\" (UniqueName: \"kubernetes.io/projected/98aa29ea-e74b-41ce-ab99-18b3b7e84338-kube-api-access-hpvzj\") pod \"monitoring-plugin-77f4f76594-tjq2x\" (UID: \"98aa29ea-e74b-41ce-ab99-18b3b7e84338\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-tjq2x" Nov 03 20:40:00.084339 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:00.084264 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7d9ffb76d9-hvc98"] Nov 03 20:40:00.086415 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:00.086391 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7d9ffb76d9-hvc98" Nov 03 20:40:00.107535 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:00.107502 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/98aa29ea-e74b-41ce-ab99-18b3b7e84338-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-tjq2x\" (UID: \"98aa29ea-e74b-41ce-ab99-18b3b7e84338\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-tjq2x" Nov 03 20:40:00.107710 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:00.107556 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpvzj\" (UniqueName: \"kubernetes.io/projected/98aa29ea-e74b-41ce-ab99-18b3b7e84338-kube-api-access-hpvzj\") pod \"monitoring-plugin-77f4f76594-tjq2x\" (UID: \"98aa29ea-e74b-41ce-ab99-18b3b7e84338\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-tjq2x" Nov 03 20:40:00.110460 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:00.110417 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/98aa29ea-e74b-41ce-ab99-18b3b7e84338-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-tjq2x\" (UID: \"98aa29ea-e74b-41ce-ab99-18b3b7e84338\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-tjq2x" Nov 03 20:40:00.141215 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:00.141179 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 03 20:40:00.141402 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:00.141307 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-6a1ucikds3fk1\"" Nov 03 20:40:00.141402 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:00.141368 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 03 20:40:00.141402 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:00.141393 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 03 20:40:00.141402 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:00.141399 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-tr5lt\"" Nov 03 20:40:00.141582 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:00.141415 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 03 20:40:00.146209 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:00.146177 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7d9ffb76d9-hvc98"] Nov 03 20:40:00.168682 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:00.168653 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpvzj\" (UniqueName: \"kubernetes.io/projected/98aa29ea-e74b-41ce-ab99-18b3b7e84338-kube-api-access-hpvzj\") pod \"monitoring-plugin-77f4f76594-tjq2x\" (UID: \"98aa29ea-e74b-41ce-ab99-18b3b7e84338\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-tjq2x" Nov 03 20:40:00.208997 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:00.208947 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7fae9b94-1aeb-4bfd-b435-4c9397907762-secret-metrics-server-tls\") pod \"metrics-server-7d9ffb76d9-hvc98\" (UID: \"7fae9b94-1aeb-4bfd-b435-4c9397907762\") " pod="openshift-monitoring/metrics-server-7d9ffb76d9-hvc98" Nov 03 20:40:00.209545 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:00.209519 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fae9b94-1aeb-4bfd-b435-4c9397907762-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7d9ffb76d9-hvc98\" (UID: \"7fae9b94-1aeb-4bfd-b435-4c9397907762\") " pod="openshift-monitoring/metrics-server-7d9ffb76d9-hvc98" Nov 03 20:40:00.209747 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:00.209726 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7fae9b94-1aeb-4bfd-b435-4c9397907762-client-ca-bundle\") pod \"metrics-server-7d9ffb76d9-hvc98\" (UID: \"7fae9b94-1aeb-4bfd-b435-4c9397907762\") " pod="openshift-monitoring/metrics-server-7d9ffb76d9-hvc98" Nov 03 20:40:00.209884 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:00.209865 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfg84\" (UniqueName: \"kubernetes.io/projected/7fae9b94-1aeb-4bfd-b435-4c9397907762-kube-api-access-xfg84\") pod \"metrics-server-7d9ffb76d9-hvc98\" (UID: \"7fae9b94-1aeb-4bfd-b435-4c9397907762\") " pod="openshift-monitoring/metrics-server-7d9ffb76d9-hvc98" Nov 03 20:40:00.209988 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:00.209971 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7fae9b94-1aeb-4bfd-b435-4c9397907762-secret-metrics-server-client-certs\") pod \"metrics-server-7d9ffb76d9-hvc98\" (UID: \"7fae9b94-1aeb-4bfd-b435-4c9397907762\") " pod="openshift-monitoring/metrics-server-7d9ffb76d9-hvc98" Nov 03 20:40:00.210124 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:00.210106 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7fae9b94-1aeb-4bfd-b435-4c9397907762-audit-log\") pod \"metrics-server-7d9ffb76d9-hvc98\" (UID: \"7fae9b94-1aeb-4bfd-b435-4c9397907762\") " pod="openshift-monitoring/metrics-server-7d9ffb76d9-hvc98" Nov 03 20:40:00.210159 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:00.210136 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7fae9b94-1aeb-4bfd-b435-4c9397907762-metrics-server-audit-profiles\") pod \"metrics-server-7d9ffb76d9-hvc98\" (UID: \"7fae9b94-1aeb-4bfd-b435-4c9397907762\") " pod="openshift-monitoring/metrics-server-7d9ffb76d9-hvc98" Nov 03 20:40:00.261239 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:00.261213 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-tjq2x" Nov 03 20:40:00.310925 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:00.310770 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7fae9b94-1aeb-4bfd-b435-4c9397907762-secret-metrics-server-tls\") pod \"metrics-server-7d9ffb76d9-hvc98\" (UID: \"7fae9b94-1aeb-4bfd-b435-4c9397907762\") " pod="openshift-monitoring/metrics-server-7d9ffb76d9-hvc98" Nov 03 20:40:00.310925 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:00.310820 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fae9b94-1aeb-4bfd-b435-4c9397907762-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7d9ffb76d9-hvc98\" (UID: \"7fae9b94-1aeb-4bfd-b435-4c9397907762\") " pod="openshift-monitoring/metrics-server-7d9ffb76d9-hvc98" Nov 03 20:40:00.310925 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:00.310868 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7fae9b94-1aeb-4bfd-b435-4c9397907762-client-ca-bundle\") pod \"metrics-server-7d9ffb76d9-hvc98\" (UID: \"7fae9b94-1aeb-4bfd-b435-4c9397907762\") " pod="openshift-monitoring/metrics-server-7d9ffb76d9-hvc98" Nov 03 20:40:00.310925 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:00.310900 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfg84\" (UniqueName: \"kubernetes.io/projected/7fae9b94-1aeb-4bfd-b435-4c9397907762-kube-api-access-xfg84\") pod \"metrics-server-7d9ffb76d9-hvc98\" (UID: \"7fae9b94-1aeb-4bfd-b435-4c9397907762\") " pod="openshift-monitoring/metrics-server-7d9ffb76d9-hvc98" Nov 03 20:40:00.311156 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:00.310956 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7fae9b94-1aeb-4bfd-b435-4c9397907762-secret-metrics-server-client-certs\") pod \"metrics-server-7d9ffb76d9-hvc98\" (UID: \"7fae9b94-1aeb-4bfd-b435-4c9397907762\") " pod="openshift-monitoring/metrics-server-7d9ffb76d9-hvc98" Nov 03 20:40:00.311156 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:00.311012 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7fae9b94-1aeb-4bfd-b435-4c9397907762-audit-log\") pod \"metrics-server-7d9ffb76d9-hvc98\" (UID: \"7fae9b94-1aeb-4bfd-b435-4c9397907762\") " pod="openshift-monitoring/metrics-server-7d9ffb76d9-hvc98" Nov 03 20:40:00.311156 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:00.311035 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7fae9b94-1aeb-4bfd-b435-4c9397907762-metrics-server-audit-profiles\") pod \"metrics-server-7d9ffb76d9-hvc98\" (UID: \"7fae9b94-1aeb-4bfd-b435-4c9397907762\") " pod="openshift-monitoring/metrics-server-7d9ffb76d9-hvc98" Nov 03 20:40:00.315848 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:00.311719 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7fae9b94-1aeb-4bfd-b435-4c9397907762-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7d9ffb76d9-hvc98\" (UID: \"7fae9b94-1aeb-4bfd-b435-4c9397907762\") " pod="openshift-monitoring/metrics-server-7d9ffb76d9-hvc98" Nov 03 20:40:00.315848 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:00.311936 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7fae9b94-1aeb-4bfd-b435-4c9397907762-audit-log\") pod \"metrics-server-7d9ffb76d9-hvc98\" (UID: \"7fae9b94-1aeb-4bfd-b435-4c9397907762\") " pod="openshift-monitoring/metrics-server-7d9ffb76d9-hvc98" Nov 03 20:40:00.315848 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:00.312806 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7fae9b94-1aeb-4bfd-b435-4c9397907762-metrics-server-audit-profiles\") pod \"metrics-server-7d9ffb76d9-hvc98\" (UID: \"7fae9b94-1aeb-4bfd-b435-4c9397907762\") " pod="openshift-monitoring/metrics-server-7d9ffb76d9-hvc98" Nov 03 20:40:00.316306 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:00.316250 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7fae9b94-1aeb-4bfd-b435-4c9397907762-client-ca-bundle\") pod \"metrics-server-7d9ffb76d9-hvc98\" (UID: \"7fae9b94-1aeb-4bfd-b435-4c9397907762\") " pod="openshift-monitoring/metrics-server-7d9ffb76d9-hvc98" Nov 03 20:40:00.317734 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:00.316844 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7fae9b94-1aeb-4bfd-b435-4c9397907762-secret-metrics-server-tls\") pod \"metrics-server-7d9ffb76d9-hvc98\" (UID: \"7fae9b94-1aeb-4bfd-b435-4c9397907762\") " pod="openshift-monitoring/metrics-server-7d9ffb76d9-hvc98" Nov 03 20:40:00.320446 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:00.320406 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7fae9b94-1aeb-4bfd-b435-4c9397907762-secret-metrics-server-client-certs\") pod \"metrics-server-7d9ffb76d9-hvc98\" (UID: \"7fae9b94-1aeb-4bfd-b435-4c9397907762\") " pod="openshift-monitoring/metrics-server-7d9ffb76d9-hvc98" Nov 03 20:40:00.372308 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:00.372236 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfg84\" (UniqueName: \"kubernetes.io/projected/7fae9b94-1aeb-4bfd-b435-4c9397907762-kube-api-access-xfg84\") pod \"metrics-server-7d9ffb76d9-hvc98\" (UID: \"7fae9b94-1aeb-4bfd-b435-4c9397907762\") " pod="openshift-monitoring/metrics-server-7d9ffb76d9-hvc98" Nov 03 20:40:00.398149 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:00.398115 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7d9ffb76d9-hvc98" Nov 03 20:40:00.628723 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:00.628646 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-tjq2x"] Nov 03 20:40:00.631786 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:40:00.631755 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98aa29ea_e74b_41ce_ab99_18b3b7e84338.slice/crio-1f293f3cf8ed80a842148ecdea550ab8488d300dc854dc94a3cdad6ce6a3a0db WatchSource:0}: Error finding container 1f293f3cf8ed80a842148ecdea550ab8488d300dc854dc94a3cdad6ce6a3a0db: Status 404 returned error can't find the container with id 1f293f3cf8ed80a842148ecdea550ab8488d300dc854dc94a3cdad6ce6a3a0db Nov 03 20:40:00.771401 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:40:00.771374 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fae9b94_1aeb_4bfd_b435_4c9397907762.slice/crio-21af16eed2294937b547f32130220f83d09fa69aa761730d06e641f3b48e1b64 WatchSource:0}: Error finding container 21af16eed2294937b547f32130220f83d09fa69aa761730d06e641f3b48e1b64: Status 404 returned error can't find the container with id 21af16eed2294937b547f32130220f83d09fa69aa761730d06e641f3b48e1b64 Nov 03 20:40:00.775354 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:00.775332 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7d9ffb76d9-hvc98"] Nov 03 20:40:00.876545 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:00.876514 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-574598d969-4hlgw"] Nov 03 20:40:00.879630 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:00.879579 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-574598d969-4hlgw" Nov 03 20:40:00.932021 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:00.931979 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 03 20:40:00.932021 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:00.932007 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 03 20:40:00.932021 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:00.932001 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 03 20:40:00.932021 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:00.932015 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 03 20:40:00.932304 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:00.932069 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 03 20:40:00.932304 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:00.932075 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-t9gn4\"" Nov 03 20:40:00.932794 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:00.932778 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 03 20:40:00.945612 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:00.945582 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-574598d969-4hlgw"] Nov 03 20:40:00.974608 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:00.974576 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-566dbff8bb-sf4f6" Nov 03 20:40:01.017951 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:01.017918 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e75299b-366a-4617-8c73-a12733d8db8c-serving-certs-ca-bundle\") pod \"telemeter-client-574598d969-4hlgw\" (UID: \"2e75299b-366a-4617-8c73-a12733d8db8c\") " pod="openshift-monitoring/telemeter-client-574598d969-4hlgw" Nov 03 20:40:01.017951 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:01.017959 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e75299b-366a-4617-8c73-a12733d8db8c-metrics-client-ca\") pod \"telemeter-client-574598d969-4hlgw\" (UID: \"2e75299b-366a-4617-8c73-a12733d8db8c\") " pod="openshift-monitoring/telemeter-client-574598d969-4hlgw" Nov 03 20:40:01.018187 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:01.017984 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2e75299b-366a-4617-8c73-a12733d8db8c-telemeter-client-tls\") pod \"telemeter-client-574598d969-4hlgw\" (UID: \"2e75299b-366a-4617-8c73-a12733d8db8c\") " pod="openshift-monitoring/telemeter-client-574598d969-4hlgw" Nov 03 20:40:01.018187 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:01.018007 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e75299b-366a-4617-8c73-a12733d8db8c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-574598d969-4hlgw\" (UID: \"2e75299b-366a-4617-8c73-a12733d8db8c\") " pod="openshift-monitoring/telemeter-client-574598d969-4hlgw" Nov 03 20:40:01.018187 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:01.018124 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2e75299b-366a-4617-8c73-a12733d8db8c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-574598d969-4hlgw\" (UID: \"2e75299b-366a-4617-8c73-a12733d8db8c\") " pod="openshift-monitoring/telemeter-client-574598d969-4hlgw" Nov 03 20:40:01.018187 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:01.018172 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2e75299b-366a-4617-8c73-a12733d8db8c-secret-telemeter-client\") pod \"telemeter-client-574598d969-4hlgw\" (UID: \"2e75299b-366a-4617-8c73-a12733d8db8c\") " pod="openshift-monitoring/telemeter-client-574598d969-4hlgw" Nov 03 20:40:01.018319 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:01.018199 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcw4f\" (UniqueName: \"kubernetes.io/projected/2e75299b-366a-4617-8c73-a12733d8db8c-kube-api-access-rcw4f\") pod \"telemeter-client-574598d969-4hlgw\" (UID: \"2e75299b-366a-4617-8c73-a12733d8db8c\") " pod="openshift-monitoring/telemeter-client-574598d969-4hlgw" Nov 03 20:40:01.018319 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:01.018219 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2e75299b-366a-4617-8c73-a12733d8db8c-federate-client-tls\") pod \"telemeter-client-574598d969-4hlgw\" (UID: \"2e75299b-366a-4617-8c73-a12733d8db8c\") " pod="openshift-monitoring/telemeter-client-574598d969-4hlgw" Nov 03 20:40:01.078204 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:01.078170 4305 generic.go:356] "Generic (PLEG): container finished" podID="5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca" containerID="3ce61a2a2fcd9fa3610ece57f88c54f9661f7e00ce09494bfc2e83db6a6bb6b9" exitCode=0 Nov 03 20:40:01.078372 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:01.078271 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca","Type":"ContainerDied","Data":"3ce61a2a2fcd9fa3610ece57f88c54f9661f7e00ce09494bfc2e83db6a6bb6b9"} Nov 03 20:40:01.079557 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:01.079535 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-tjq2x" event={"ID":"98aa29ea-e74b-41ce-ab99-18b3b7e84338","Type":"ContainerStarted","Data":"1f293f3cf8ed80a842148ecdea550ab8488d300dc854dc94a3cdad6ce6a3a0db"} Nov 03 20:40:01.080644 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:01.080629 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7d9ffb76d9-hvc98" event={"ID":"7fae9b94-1aeb-4bfd-b435-4c9397907762","Type":"ContainerStarted","Data":"21af16eed2294937b547f32130220f83d09fa69aa761730d06e641f3b48e1b64"} Nov 03 20:40:01.118981 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:01.118943 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e75299b-366a-4617-8c73-a12733d8db8c-serving-certs-ca-bundle\") pod \"telemeter-client-574598d969-4hlgw\" (UID: \"2e75299b-366a-4617-8c73-a12733d8db8c\") " pod="openshift-monitoring/telemeter-client-574598d969-4hlgw" Nov 03 20:40:01.119494 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:01.118992 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e75299b-366a-4617-8c73-a12733d8db8c-metrics-client-ca\") pod \"telemeter-client-574598d969-4hlgw\" (UID: \"2e75299b-366a-4617-8c73-a12733d8db8c\") " pod="openshift-monitoring/telemeter-client-574598d969-4hlgw" Nov 03 20:40:01.119494 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:01.119040 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2e75299b-366a-4617-8c73-a12733d8db8c-telemeter-client-tls\") pod \"telemeter-client-574598d969-4hlgw\" (UID: \"2e75299b-366a-4617-8c73-a12733d8db8c\") " pod="openshift-monitoring/telemeter-client-574598d969-4hlgw" Nov 03 20:40:01.119494 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:01.119073 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e75299b-366a-4617-8c73-a12733d8db8c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-574598d969-4hlgw\" (UID: \"2e75299b-366a-4617-8c73-a12733d8db8c\") " pod="openshift-monitoring/telemeter-client-574598d969-4hlgw" Nov 03 20:40:01.119494 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:01.119106 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2e75299b-366a-4617-8c73-a12733d8db8c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-574598d969-4hlgw\" (UID: \"2e75299b-366a-4617-8c73-a12733d8db8c\") " pod="openshift-monitoring/telemeter-client-574598d969-4hlgw" Nov 03 20:40:01.119494 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:01.119137 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2e75299b-366a-4617-8c73-a12733d8db8c-secret-telemeter-client\") pod \"telemeter-client-574598d969-4hlgw\" (UID: \"2e75299b-366a-4617-8c73-a12733d8db8c\") " pod="openshift-monitoring/telemeter-client-574598d969-4hlgw" Nov 03 20:40:01.119494 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:01.119194 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcw4f\" (UniqueName: \"kubernetes.io/projected/2e75299b-366a-4617-8c73-a12733d8db8c-kube-api-access-rcw4f\") pod \"telemeter-client-574598d969-4hlgw\" (UID: \"2e75299b-366a-4617-8c73-a12733d8db8c\") " pod="openshift-monitoring/telemeter-client-574598d969-4hlgw" Nov 03 20:40:01.119494 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:01.119247 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2e75299b-366a-4617-8c73-a12733d8db8c-federate-client-tls\") pod \"telemeter-client-574598d969-4hlgw\" (UID: \"2e75299b-366a-4617-8c73-a12733d8db8c\") " pod="openshift-monitoring/telemeter-client-574598d969-4hlgw" Nov 03 20:40:01.120814 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:01.120776 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e75299b-366a-4617-8c73-a12733d8db8c-serving-certs-ca-bundle\") pod \"telemeter-client-574598d969-4hlgw\" (UID: \"2e75299b-366a-4617-8c73-a12733d8db8c\") " pod="openshift-monitoring/telemeter-client-574598d969-4hlgw" Nov 03 20:40:01.120814 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:01.120787 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e75299b-366a-4617-8c73-a12733d8db8c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-574598d969-4hlgw\" (UID: \"2e75299b-366a-4617-8c73-a12733d8db8c\") " pod="openshift-monitoring/telemeter-client-574598d969-4hlgw" Nov 03 20:40:01.121501 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:01.121482 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e75299b-366a-4617-8c73-a12733d8db8c-metrics-client-ca\") pod \"telemeter-client-574598d969-4hlgw\" (UID: \"2e75299b-366a-4617-8c73-a12733d8db8c\") " pod="openshift-monitoring/telemeter-client-574598d969-4hlgw" Nov 03 20:40:01.122823 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:01.122803 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2e75299b-366a-4617-8c73-a12733d8db8c-telemeter-client-tls\") pod \"telemeter-client-574598d969-4hlgw\" (UID: \"2e75299b-366a-4617-8c73-a12733d8db8c\") " pod="openshift-monitoring/telemeter-client-574598d969-4hlgw" Nov 03 20:40:01.122951 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:01.122877 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2e75299b-366a-4617-8c73-a12733d8db8c-federate-client-tls\") pod \"telemeter-client-574598d969-4hlgw\" (UID: \"2e75299b-366a-4617-8c73-a12733d8db8c\") " pod="openshift-monitoring/telemeter-client-574598d969-4hlgw" Nov 03 20:40:01.123222 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:01.123202 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2e75299b-366a-4617-8c73-a12733d8db8c-secret-telemeter-client\") pod \"telemeter-client-574598d969-4hlgw\" (UID: \"2e75299b-366a-4617-8c73-a12733d8db8c\") " pod="openshift-monitoring/telemeter-client-574598d969-4hlgw" Nov 03 20:40:01.123615 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:01.123595 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2e75299b-366a-4617-8c73-a12733d8db8c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-574598d969-4hlgw\" (UID: \"2e75299b-366a-4617-8c73-a12733d8db8c\") " pod="openshift-monitoring/telemeter-client-574598d969-4hlgw" Nov 03 20:40:01.182568 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:01.182487 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcw4f\" (UniqueName: \"kubernetes.io/projected/2e75299b-366a-4617-8c73-a12733d8db8c-kube-api-access-rcw4f\") pod \"telemeter-client-574598d969-4hlgw\" (UID: \"2e75299b-366a-4617-8c73-a12733d8db8c\") " pod="openshift-monitoring/telemeter-client-574598d969-4hlgw" Nov 03 20:40:01.192141 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:01.192113 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-574598d969-4hlgw" Nov 03 20:40:01.374656 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:01.374620 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-66c6bd48db-bjx28" Nov 03 20:40:01.374840 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:01.374690 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-66c6bd48db-bjx28" Nov 03 20:40:01.379516 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:01.379494 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-66c6bd48db-bjx28" Nov 03 20:40:01.571604 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:01.571533 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-574598d969-4hlgw"] Nov 03 20:40:01.574703 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:40:01.574676 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e75299b_366a_4617_8c73_a12733d8db8c.slice/crio-4d4f0e723512ee0bce92ef1305648e2d59dad32019cc536afac088fe7e875ff6 WatchSource:0}: Error finding container 4d4f0e723512ee0bce92ef1305648e2d59dad32019cc536afac088fe7e875ff6: Status 404 returned error can't find the container with id 4d4f0e723512ee0bce92ef1305648e2d59dad32019cc536afac088fe7e875ff6 Nov 03 20:40:02.094054 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:02.094011 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-574598d969-4hlgw" event={"ID":"2e75299b-366a-4617-8c73-a12733d8db8c","Type":"ContainerStarted","Data":"4d4f0e723512ee0bce92ef1305648e2d59dad32019cc536afac088fe7e875ff6"} Nov 03 20:40:02.099673 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:02.099644 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-66c6bd48db-bjx28" Nov 03 20:40:03.100864 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:03.100828 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7d9ffb76d9-hvc98" event={"ID":"7fae9b94-1aeb-4bfd-b435-4c9397907762","Type":"ContainerStarted","Data":"2a6f03cb8c771587c36eafac389abc235f897d82da9bba0c34de969d40fc228b"} Nov 03 20:40:03.104370 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:03.104336 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-tjq2x" event={"ID":"98aa29ea-e74b-41ce-ab99-18b3b7e84338","Type":"ContainerStarted","Data":"768e50a73af60405f0e19dacc8e2000a8d3fe973318b6c1688a3c4f9c7412bb8"} Nov 03 20:40:03.104536 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:03.104403 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-tjq2x" Nov 03 20:40:03.105925 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:03.105904 4305 patch_prober.go:29] interesting pod/monitoring-plugin-77f4f76594-tjq2x container/monitoring-plugin namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.14:9443/health\": dial tcp 10.134.0.14:9443: connect: connection refused" start-of-body= Nov 03 20:40:03.106036 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:03.105945 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/monitoring-plugin-77f4f76594-tjq2x" podUID="98aa29ea-e74b-41ce-ab99-18b3b7e84338" containerName="monitoring-plugin" probeResult="failure" output="Get \"https://10.134.0.14:9443/health\": dial tcp 10.134.0.14:9443: connect: connection refused" Nov 03 20:40:03.218853 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:03.218750 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7d9ffb76d9-hvc98" podStartSLOduration=1.172158062 podStartE2EDuration="3.21873194s" totalImagesPullingTime="2.046573878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:40:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:40:03.217281109 +0000 UTC m=+38.442600925" watchObservedRunningTime="2026-11-03 20:40:03.21873194 +0000 UTC m=+38.444051733" Nov 03 20:40:03.333394 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:03.333289 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-tjq2x" podStartSLOduration=1.9499983090000002 podStartE2EDuration="4.33327577s" totalImagesPullingTime="2.383277461s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:39:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:40:03.332076459 +0000 UTC m=+38.557396250" watchObservedRunningTime="2026-11-03 20:40:03.33327577 +0000 UTC m=+38.558595561" Nov 03 20:40:04.110278 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.110183 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-574598d969-4hlgw" event={"ID":"2e75299b-366a-4617-8c73-a12733d8db8c","Type":"ContainerStarted","Data":"72bf7e1072d7ce2e5a72c2a7c702c605b610f891f9381fdffda1bc9aef8ae6f2"} Nov 03 20:40:04.110278 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.110251 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-574598d969-4hlgw" event={"ID":"2e75299b-366a-4617-8c73-a12733d8db8c","Type":"ContainerStarted","Data":"d828ab3ce704627668ecf6e9c3c95bc389fbc48caf9283803cdd11f2dbbc2b12"} Nov 03 20:40:04.110278 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.110278 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-574598d969-4hlgw" event={"ID":"2e75299b-366a-4617-8c73-a12733d8db8c","Type":"ContainerStarted","Data":"f9a6ffa1ec1213b14e3554bbae77866bb84a4515a7aa30d9b803147cadc38d39"} Nov 03 20:40:04.114872 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.114845 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-tjq2x" Nov 03 20:40:04.212765 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.212730 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5b675ff864-tw7z9"] Nov 03 20:40:04.215111 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.215092 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b675ff864-tw7z9" Nov 03 20:40:04.319693 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.319658 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 03 20:40:04.350228 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.350198 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13737861-083d-4ed4-bdb5-2022863b54d4-trusted-ca-bundle\") pod \"console-5b675ff864-tw7z9\" (UID: \"13737861-083d-4ed4-bdb5-2022863b54d4\") " pod="openshift-console/console-5b675ff864-tw7z9" Nov 03 20:40:04.350397 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.350239 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/13737861-083d-4ed4-bdb5-2022863b54d4-oauth-serving-cert\") pod \"console-5b675ff864-tw7z9\" (UID: \"13737861-083d-4ed4-bdb5-2022863b54d4\") " pod="openshift-console/console-5b675ff864-tw7z9" Nov 03 20:40:04.350397 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.350263 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/13737861-083d-4ed4-bdb5-2022863b54d4-console-oauth-config\") pod \"console-5b675ff864-tw7z9\" (UID: \"13737861-083d-4ed4-bdb5-2022863b54d4\") " pod="openshift-console/console-5b675ff864-tw7z9" Nov 03 20:40:04.350397 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.350323 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6b5t\" (UniqueName: \"kubernetes.io/projected/13737861-083d-4ed4-bdb5-2022863b54d4-kube-api-access-q6b5t\") pod \"console-5b675ff864-tw7z9\" (UID: \"13737861-083d-4ed4-bdb5-2022863b54d4\") " pod="openshift-console/console-5b675ff864-tw7z9" Nov 03 20:40:04.350582 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.350398 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/13737861-083d-4ed4-bdb5-2022863b54d4-tmp\") pod \"console-5b675ff864-tw7z9\" (UID: \"13737861-083d-4ed4-bdb5-2022863b54d4\") " pod="openshift-console/console-5b675ff864-tw7z9" Nov 03 20:40:04.350582 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.350488 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/13737861-083d-4ed4-bdb5-2022863b54d4-console-serving-cert\") pod \"console-5b675ff864-tw7z9\" (UID: \"13737861-083d-4ed4-bdb5-2022863b54d4\") " pod="openshift-console/console-5b675ff864-tw7z9" Nov 03 20:40:04.350582 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.350543 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/13737861-083d-4ed4-bdb5-2022863b54d4-service-ca\") pod \"console-5b675ff864-tw7z9\" (UID: \"13737861-083d-4ed4-bdb5-2022863b54d4\") " pod="openshift-console/console-5b675ff864-tw7z9" Nov 03 20:40:04.350684 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.350594 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/13737861-083d-4ed4-bdb5-2022863b54d4-console-config\") pod \"console-5b675ff864-tw7z9\" (UID: \"13737861-083d-4ed4-bdb5-2022863b54d4\") " pod="openshift-console/console-5b675ff864-tw7z9" Nov 03 20:40:04.369504 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.369456 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-574598d969-4hlgw" podStartSLOduration=2.115565666 podStartE2EDuration="4.369415413s" totalImagesPullingTime="2.253849747s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:40:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:40:04.320725562 +0000 UTC m=+39.546045354" watchObservedRunningTime="2026-11-03 20:40:04.369415413 +0000 UTC m=+39.594735204" Nov 03 20:40:04.370891 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.370871 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b675ff864-tw7z9"] Nov 03 20:40:04.371865 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.371849 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 20:40:04.374756 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.374740 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:04.427444 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.427400 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 03 20:40:04.430588 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.430567 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 03 20:40:04.431549 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.431531 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 03 20:40:04.431682 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.431543 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 03 20:40:04.431682 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.431543 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 03 20:40:04.431682 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.431542 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 03 20:40:04.431890 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.431550 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 03 20:40:04.431890 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.431657 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-29m4i5anvmnkl\"" Nov 03 20:40:04.453509 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.452114 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:04.453509 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.452168 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-web-config\") pod \"prometheus-k8s-0\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:04.453509 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.452191 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:04.453509 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.452223 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6b5t\" (UniqueName: \"kubernetes.io/projected/13737861-083d-4ed4-bdb5-2022863b54d4-kube-api-access-q6b5t\") pod \"console-5b675ff864-tw7z9\" (UID: \"13737861-083d-4ed4-bdb5-2022863b54d4\") " pod="openshift-console/console-5b675ff864-tw7z9" Nov 03 20:40:04.453509 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.452247 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:04.453509 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.452271 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:04.453509 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.452295 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:04.453509 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.452333 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/13737861-083d-4ed4-bdb5-2022863b54d4-tmp\") pod \"console-5b675ff864-tw7z9\" (UID: \"13737861-083d-4ed4-bdb5-2022863b54d4\") " pod="openshift-console/console-5b675ff864-tw7z9" Nov 03 20:40:04.453509 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.452355 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:04.453509 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.452373 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:04.453509 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.452404 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/13737861-083d-4ed4-bdb5-2022863b54d4-console-serving-cert\") pod \"console-5b675ff864-tw7z9\" (UID: \"13737861-083d-4ed4-bdb5-2022863b54d4\") " pod="openshift-console/console-5b675ff864-tw7z9" Nov 03 20:40:04.453509 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.453127 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/13737861-083d-4ed4-bdb5-2022863b54d4-tmp\") pod \"console-5b675ff864-tw7z9\" (UID: \"13737861-083d-4ed4-bdb5-2022863b54d4\") " pod="openshift-console/console-5b675ff864-tw7z9" Nov 03 20:40:04.453509 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.452454 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:04.453509 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.453514 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:04.454243 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.453559 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/13737861-083d-4ed4-bdb5-2022863b54d4-service-ca\") pod \"console-5b675ff864-tw7z9\" (UID: \"13737861-083d-4ed4-bdb5-2022863b54d4\") " pod="openshift-console/console-5b675ff864-tw7z9" Nov 03 20:40:04.454243 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.453583 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-config\") pod \"prometheus-k8s-0\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:04.454243 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.453607 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:04.454243 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.453641 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/13737861-083d-4ed4-bdb5-2022863b54d4-console-config\") pod \"console-5b675ff864-tw7z9\" (UID: \"13737861-083d-4ed4-bdb5-2022863b54d4\") " pod="openshift-console/console-5b675ff864-tw7z9" Nov 03 20:40:04.454243 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.453668 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:04.454243 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.453703 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:04.454243 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.453747 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13737861-083d-4ed4-bdb5-2022863b54d4-trusted-ca-bundle\") pod \"console-5b675ff864-tw7z9\" (UID: \"13737861-083d-4ed4-bdb5-2022863b54d4\") " pod="openshift-console/console-5b675ff864-tw7z9" Nov 03 20:40:04.454243 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.453776 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:04.454243 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.453802 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:04.454243 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.453832 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/13737861-083d-4ed4-bdb5-2022863b54d4-oauth-serving-cert\") pod \"console-5b675ff864-tw7z9\" (UID: \"13737861-083d-4ed4-bdb5-2022863b54d4\") " pod="openshift-console/console-5b675ff864-tw7z9" Nov 03 20:40:04.454243 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.453857 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:04.454243 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.453888 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h84lh\" (UniqueName: \"kubernetes.io/projected/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-kube-api-access-h84lh\") pod \"prometheus-k8s-0\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:04.454243 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.453931 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/13737861-083d-4ed4-bdb5-2022863b54d4-console-oauth-config\") pod \"console-5b675ff864-tw7z9\" (UID: \"13737861-083d-4ed4-bdb5-2022863b54d4\") " pod="openshift-console/console-5b675ff864-tw7z9" Nov 03 20:40:04.454243 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.453954 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-config-out\") pod \"prometheus-k8s-0\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:04.454243 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.453980 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:04.455018 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.454756 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/13737861-083d-4ed4-bdb5-2022863b54d4-oauth-serving-cert\") pod \"console-5b675ff864-tw7z9\" (UID: \"13737861-083d-4ed4-bdb5-2022863b54d4\") " pod="openshift-console/console-5b675ff864-tw7z9" Nov 03 20:40:04.455018 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.454922 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13737861-083d-4ed4-bdb5-2022863b54d4-trusted-ca-bundle\") pod \"console-5b675ff864-tw7z9\" (UID: \"13737861-083d-4ed4-bdb5-2022863b54d4\") " pod="openshift-console/console-5b675ff864-tw7z9" Nov 03 20:40:04.455637 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.455237 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/13737861-083d-4ed4-bdb5-2022863b54d4-service-ca\") pod \"console-5b675ff864-tw7z9\" (UID: \"13737861-083d-4ed4-bdb5-2022863b54d4\") " pod="openshift-console/console-5b675ff864-tw7z9" Nov 03 20:40:04.455637 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.455550 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/13737861-083d-4ed4-bdb5-2022863b54d4-console-config\") pod \"console-5b675ff864-tw7z9\" (UID: \"13737861-083d-4ed4-bdb5-2022863b54d4\") " pod="openshift-console/console-5b675ff864-tw7z9" Nov 03 20:40:04.456504 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.456073 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/13737861-083d-4ed4-bdb5-2022863b54d4-console-serving-cert\") pod \"console-5b675ff864-tw7z9\" (UID: \"13737861-083d-4ed4-bdb5-2022863b54d4\") " pod="openshift-console/console-5b675ff864-tw7z9" Nov 03 20:40:04.457590 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.457565 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/13737861-083d-4ed4-bdb5-2022863b54d4-console-oauth-config\") pod \"console-5b675ff864-tw7z9\" (UID: \"13737861-083d-4ed4-bdb5-2022863b54d4\") " pod="openshift-console/console-5b675ff864-tw7z9" Nov 03 20:40:04.458083 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.458063 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 20:40:04.513545 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.513473 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6b5t\" (UniqueName: \"kubernetes.io/projected/13737861-083d-4ed4-bdb5-2022863b54d4-kube-api-access-q6b5t\") pod \"console-5b675ff864-tw7z9\" (UID: \"13737861-083d-4ed4-bdb5-2022863b54d4\") " pod="openshift-console/console-5b675ff864-tw7z9" Nov 03 20:40:04.523203 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.523178 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 03 20:40:04.523203 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.523197 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 03 20:40:04.523371 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.523216 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 03 20:40:04.524205 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.524187 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 03 20:40:04.531186 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.531162 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 03 20:40:04.531302 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.531188 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-vj5gx\"" Nov 03 20:40:04.531302 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.531216 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 03 20:40:04.531302 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.531260 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 03 20:40:04.540576 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.540558 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b675ff864-tw7z9" Nov 03 20:40:04.554976 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.554953 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-config\") pod \"prometheus-k8s-0\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:04.555061 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.554995 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:04.555061 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.555033 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:04.555167 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.555065 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:04.555167 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.555107 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:04.555167 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.555130 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:04.555167 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.555158 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:04.555363 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.555185 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h84lh\" (UniqueName: \"kubernetes.io/projected/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-kube-api-access-h84lh\") pod \"prometheus-k8s-0\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:04.555363 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.555223 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-config-out\") pod \"prometheus-k8s-0\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:04.555363 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.555245 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:04.555363 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.555267 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:04.555363 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.555302 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-web-config\") pod \"prometheus-k8s-0\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:04.555363 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.555324 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:04.555363 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.555352 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:04.555754 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.555375 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:04.555754 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.555401 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:04.555754 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.555462 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:04.555754 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.555492 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:04.555754 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.555532 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:04.555754 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.555574 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:04.556242 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.556081 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:04.556572 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.556379 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:04.556572 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.556420 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:04.558803 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.558774 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:04.560012 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.559967 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:04.561137 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.561112 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:04.563316 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.561684 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-web-config\") pod \"prometheus-k8s-0\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:04.563316 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.562308 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:04.563316 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.563143 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:04.563316 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.563281 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:04.564673 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.564647 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-config-out\") pod \"prometheus-k8s-0\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:04.565565 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.565542 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:04.566269 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.565811 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:04.566269 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.566219 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-config\") pod \"prometheus-k8s-0\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:04.566269 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.566227 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:04.566969 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.566653 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:04.566969 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.566938 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:04.567815 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.567773 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:04.568815 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.568758 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:04.619855 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.619798 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h84lh\" (UniqueName: \"kubernetes.io/projected/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-kube-api-access-h84lh\") pod \"prometheus-k8s-0\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:04.701907 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.701864 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:04.927735 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:04.922677 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b675ff864-tw7z9"] Nov 03 20:40:05.117967 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:05.117929 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca","Type":"ContainerStarted","Data":"4feeac0fe9271e272a54f5602044c10d9fb124b42ab2eb0c6174df01b15ae09c"} Nov 03 20:40:05.118396 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:05.118031 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca","Type":"ContainerStarted","Data":"40cb4d8e08c1bd2ac44a28a2720f3b5fc6acb0900185aa0198325e82120dbcc3"} Nov 03 20:40:05.118396 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:05.118060 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca","Type":"ContainerStarted","Data":"299d01664c3913b3d5da14b18b9d43f1b35e89998919279d8890f9d368166849"} Nov 03 20:40:05.118396 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:05.118082 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca","Type":"ContainerStarted","Data":"08214c61e55f9633bf07868bd749eec5e7c72e8d7cd9ade9a4c06659c45a4040"} Nov 03 20:40:05.118396 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:05.118112 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca","Type":"ContainerStarted","Data":"cf0c1e86c1d12db1ba181f0a4f0df234569f4c716d96569b4dcc8ed9c2a3bdee"} Nov 03 20:40:05.119478 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:05.119449 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b675ff864-tw7z9" event={"ID":"13737861-083d-4ed4-bdb5-2022863b54d4","Type":"ContainerStarted","Data":"2851f75a21585a48c3b8b67f4a4291ce7f0390a9833737f3da909f2f6f32255c"} Nov 03 20:40:05.119619 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:05.119495 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b675ff864-tw7z9" event={"ID":"13737861-083d-4ed4-bdb5-2022863b54d4","Type":"ContainerStarted","Data":"878983fd4211468704c61e2946c638f4288b40f56c1658ddbbfc06b13bd9c5f9"} Nov 03 20:40:05.134302 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:05.134276 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 20:40:05.137270 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:40:05.137244 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34de3dcc_9aa3_4c4a_b827_d8e944c3514c.slice/crio-417ede539e89a9735cbab37d491457783b95059daabbf50e1c7c6060c63e16d4 WatchSource:0}: Error finding container 417ede539e89a9735cbab37d491457783b95059daabbf50e1c7c6060c63e16d4: Status 404 returned error can't find the container with id 417ede539e89a9735cbab37d491457783b95059daabbf50e1c7c6060c63e16d4 Nov 03 20:40:05.248227 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:05.248163 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5b675ff864-tw7z9" podStartSLOduration=1.248144534 podStartE2EDuration="1.248144534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:40:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:40:05.247414706 +0000 UTC m=+40.472734496" watchObservedRunningTime="2026-11-03 20:40:05.248144534 +0000 UTC m=+40.473464325" Nov 03 20:40:06.125244 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:06.125148 4305 generic.go:356] "Generic (PLEG): container finished" podID="34de3dcc-9aa3-4c4a-b827-d8e944c3514c" containerID="efadbb2d8cc43881c639f3892f83dfb258447b4c5292791e3e4670c1b83a68a8" exitCode=0 Nov 03 20:40:06.125742 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:06.125232 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"34de3dcc-9aa3-4c4a-b827-d8e944c3514c","Type":"ContainerDied","Data":"efadbb2d8cc43881c639f3892f83dfb258447b4c5292791e3e4670c1b83a68a8"} Nov 03 20:40:06.125742 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:06.125313 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"34de3dcc-9aa3-4c4a-b827-d8e944c3514c","Type":"ContainerStarted","Data":"417ede539e89a9735cbab37d491457783b95059daabbf50e1c7c6060c63e16d4"} Nov 03 20:40:06.129015 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:06.128986 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca","Type":"ContainerStarted","Data":"5858c44f50e2e7cec866c9064cb861cee92f8998b3c4922c3d80bcb7a8793a92"} Nov 03 20:40:06.385733 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:06.385586 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.224868671 podStartE2EDuration="9.385566615s" totalImagesPullingTime="6.147261142s" totalInitContainerRuntime="13.436802ms" isStatefulPod=false podCreationTimestamp="2026-11-03 20:39:57 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:40:06.384781803 +0000 UTC m=+41.610101634" watchObservedRunningTime="2026-11-03 20:40:06.385566615 +0000 UTC m=+41.610886407" Nov 03 20:40:11.152602 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:11.152564 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"34de3dcc-9aa3-4c4a-b827-d8e944c3514c","Type":"ContainerStarted","Data":"60ac802a7d17285286485bda07751fb435346c744067be63b6fe15f16d6fd0ef"} Nov 03 20:40:11.152995 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:11.152631 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"34de3dcc-9aa3-4c4a-b827-d8e944c3514c","Type":"ContainerStarted","Data":"1c7b0642697af1976a78b786dcf46e1278263bd5d94362657b2908d14d735b59"} Nov 03 20:40:14.166214 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:14.166183 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"34de3dcc-9aa3-4c4a-b827-d8e944c3514c","Type":"ContainerStarted","Data":"a45db29215d088d64aceff9ecab04a73618d50c292f4457f2bc5aa96f78d8ac1"} Nov 03 20:40:14.166714 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:14.166254 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"34de3dcc-9aa3-4c4a-b827-d8e944c3514c","Type":"ContainerStarted","Data":"9924a3acd9fb2dcf053317e355d5e0174fe334929f22a3b18991f251916c34bb"} Nov 03 20:40:14.166714 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:14.166283 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"34de3dcc-9aa3-4c4a-b827-d8e944c3514c","Type":"ContainerStarted","Data":"d09c16868fd4bc2d1c618abf0d5538842aef98d14056fc9cabe0d320f373ea55"} Nov 03 20:40:14.166714 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:14.166307 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"34de3dcc-9aa3-4c4a-b827-d8e944c3514c","Type":"ContainerStarted","Data":"0a743ec9b02a2c07e921055a28c26271f7f651cdab2a4120d66794950ebb731e"} Nov 03 20:40:14.426080 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:14.425972 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.829084972 podStartE2EDuration="10.425957788s" totalImagesPullingTime="7.580695399s" totalInitContainerRuntime="16.177417ms" isStatefulPod=false podCreationTimestamp="2026-11-03 20:40:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:40:14.425112719 +0000 UTC m=+49.650432515" watchObservedRunningTime="2026-11-03 20:40:14.425957788 +0000 UTC m=+49.651277578" Nov 03 20:40:14.541342 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:14.541310 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5b675ff864-tw7z9" Nov 03 20:40:14.541547 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:14.541373 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5b675ff864-tw7z9" Nov 03 20:40:14.546228 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:14.546209 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5b675ff864-tw7z9" Nov 03 20:40:14.703040 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:14.702954 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:15.174043 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:15.174008 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5b675ff864-tw7z9" Nov 03 20:40:15.322292 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:15.322261 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66c6bd48db-bjx28"] Nov 03 20:40:16.321127 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:40:16.321084 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 20:40:20.398892 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:20.398838 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7d9ffb76d9-hvc98" Nov 03 20:40:20.399379 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:20.398919 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7d9ffb76d9-hvc98" Nov 03 20:40:21.143118 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:21.143085 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:21.162645 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:21.162619 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:21.209050 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:21.209025 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:26.892701 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:26.892640 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b675ff864-tw7z9"] Nov 03 20:40:40.343169 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:40.343121 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-66c6bd48db-bjx28" podUID="33077f68-6503-41d4-a597-4f2b403b9340" containerName="console" containerID="cri-o://4ddba416c5b5733368960574da821a5e7a935ec132c5a5617163be3fdcf19dcb" gracePeriod=15 Nov 03 20:40:40.403708 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:40.403681 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7d9ffb76d9-hvc98" Nov 03 20:40:40.407599 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:40.407577 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7d9ffb76d9-hvc98" Nov 03 20:40:40.617209 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:40.617181 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66c6bd48db-bjx28" Nov 03 20:40:40.692100 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:40.692069 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33077f68-6503-41d4-a597-4f2b403b9340-console-config\" (UniqueName: \"kubernetes.io/configmap/33077f68-6503-41d4-a597-4f2b403b9340-console-config\") pod \"33077f68-6503-41d4-a597-4f2b403b9340\" (UID: \"33077f68-6503-41d4-a597-4f2b403b9340\") " Nov 03 20:40:40.692258 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:40.692131 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33077f68-6503-41d4-a597-4f2b403b9340-service-ca\" (UniqueName: \"kubernetes.io/configmap/33077f68-6503-41d4-a597-4f2b403b9340-service-ca\") pod \"33077f68-6503-41d4-a597-4f2b403b9340\" (UID: \"33077f68-6503-41d4-a597-4f2b403b9340\") " Nov 03 20:40:40.692258 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:40.692188 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33077f68-6503-41d4-a597-4f2b403b9340-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/33077f68-6503-41d4-a597-4f2b403b9340-console-oauth-config\") pod \"33077f68-6503-41d4-a597-4f2b403b9340\" (UID: \"33077f68-6503-41d4-a597-4f2b403b9340\") " Nov 03 20:40:40.692258 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:40.692216 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33077f68-6503-41d4-a597-4f2b403b9340-kube-api-access-g2rqq\" (UniqueName: \"kubernetes.io/projected/33077f68-6503-41d4-a597-4f2b403b9340-kube-api-access-g2rqq\") pod \"33077f68-6503-41d4-a597-4f2b403b9340\" (UID: \"33077f68-6503-41d4-a597-4f2b403b9340\") " Nov 03 20:40:40.692258 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:40.692238 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33077f68-6503-41d4-a597-4f2b403b9340-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/33077f68-6503-41d4-a597-4f2b403b9340-oauth-serving-cert\") pod \"33077f68-6503-41d4-a597-4f2b403b9340\" (UID: \"33077f68-6503-41d4-a597-4f2b403b9340\") " Nov 03 20:40:40.692525 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:40.692278 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33077f68-6503-41d4-a597-4f2b403b9340-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/33077f68-6503-41d4-a597-4f2b403b9340-console-serving-cert\") pod \"33077f68-6503-41d4-a597-4f2b403b9340\" (UID: \"33077f68-6503-41d4-a597-4f2b403b9340\") " Nov 03 20:40:40.692525 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:40.692307 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33077f68-6503-41d4-a597-4f2b403b9340-tmp\" (UniqueName: \"kubernetes.io/empty-dir/33077f68-6503-41d4-a597-4f2b403b9340-tmp\") pod \"33077f68-6503-41d4-a597-4f2b403b9340\" (UID: \"33077f68-6503-41d4-a597-4f2b403b9340\") " Nov 03 20:40:40.692682 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:40.692655 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33077f68-6503-41d4-a597-4f2b403b9340-service-ca" pod "33077f68-6503-41d4-a597-4f2b403b9340" (UID: "33077f68-6503-41d4-a597-4f2b403b9340"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:40:40.692876 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:40.692854 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33077f68-6503-41d4-a597-4f2b403b9340-console-config" pod "33077f68-6503-41d4-a597-4f2b403b9340" (UID: "33077f68-6503-41d4-a597-4f2b403b9340"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:40:40.692876 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:40.692751 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33077f68-6503-41d4-a597-4f2b403b9340-tmp" pod "33077f68-6503-41d4-a597-4f2b403b9340" (UID: "33077f68-6503-41d4-a597-4f2b403b9340"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:40.692876 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:40.692778 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33077f68-6503-41d4-a597-4f2b403b9340-oauth-serving-cert" pod "33077f68-6503-41d4-a597-4f2b403b9340" (UID: "33077f68-6503-41d4-a597-4f2b403b9340"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:40:40.694681 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:40.694649 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33077f68-6503-41d4-a597-4f2b403b9340-console-oauth-config" pod "33077f68-6503-41d4-a597-4f2b403b9340" (UID: "33077f68-6503-41d4-a597-4f2b403b9340"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:40:40.694770 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:40.694691 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33077f68-6503-41d4-a597-4f2b403b9340-kube-api-access-g2rqq" pod "33077f68-6503-41d4-a597-4f2b403b9340" (UID: "33077f68-6503-41d4-a597-4f2b403b9340"). InnerVolumeSpecName "kube-api-access-g2rqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:40.694770 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:40.694730 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33077f68-6503-41d4-a597-4f2b403b9340-console-serving-cert" pod "33077f68-6503-41d4-a597-4f2b403b9340" (UID: "33077f68-6503-41d4-a597-4f2b403b9340"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:40:40.793332 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:40.793294 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2rqq\" (UniqueName: \"kubernetes.io/projected/33077f68-6503-41d4-a597-4f2b403b9340-kube-api-access-g2rqq\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:40:40.793332 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:40.793324 4305 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/33077f68-6503-41d4-a597-4f2b403b9340-oauth-serving-cert\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:40:40.793332 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:40.793334 4305 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/33077f68-6503-41d4-a597-4f2b403b9340-console-serving-cert\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:40:40.793599 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:40.793343 4305 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/33077f68-6503-41d4-a597-4f2b403b9340-tmp\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:40:40.793599 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:40.793355 4305 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/33077f68-6503-41d4-a597-4f2b403b9340-console-config\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:40:40.793599 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:40.793363 4305 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/33077f68-6503-41d4-a597-4f2b403b9340-service-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:40:40.793599 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:40.793371 4305 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/33077f68-6503-41d4-a597-4f2b403b9340-console-oauth-config\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:40:41.268355 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:41.268263 4305 generic.go:356] "Generic (PLEG): container finished" podID="33077f68-6503-41d4-a597-4f2b403b9340" containerID="4ddba416c5b5733368960574da821a5e7a935ec132c5a5617163be3fdcf19dcb" exitCode=0 Nov 03 20:40:41.268355 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:41.268312 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66c6bd48db-bjx28" Nov 03 20:40:41.268355 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:41.268346 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66c6bd48db-bjx28" event={"ID":"33077f68-6503-41d4-a597-4f2b403b9340","Type":"ContainerDied","Data":"4ddba416c5b5733368960574da821a5e7a935ec132c5a5617163be3fdcf19dcb"} Nov 03 20:40:41.268661 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:41.268407 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66c6bd48db-bjx28" event={"ID":"33077f68-6503-41d4-a597-4f2b403b9340","Type":"ContainerDied","Data":"110a70d3cc183bad7461152cc326f46fa69683de22ed2e80cd728eec5209f272"} Nov 03 20:40:41.268661 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:41.268471 4305 scope.go:122] "RemoveContainer" containerID="4ddba416c5b5733368960574da821a5e7a935ec132c5a5617163be3fdcf19dcb" Nov 03 20:40:41.278733 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:41.278716 4305 scope.go:122] "RemoveContainer" containerID="4ddba416c5b5733368960574da821a5e7a935ec132c5a5617163be3fdcf19dcb" Nov 03 20:40:41.279043 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:40:41.279020 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ddba416c5b5733368960574da821a5e7a935ec132c5a5617163be3fdcf19dcb\": container with ID starting with 4ddba416c5b5733368960574da821a5e7a935ec132c5a5617163be3fdcf19dcb not found: ID does not exist" containerID="4ddba416c5b5733368960574da821a5e7a935ec132c5a5617163be3fdcf19dcb" Nov 03 20:40:41.279092 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:41.279055 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ddba416c5b5733368960574da821a5e7a935ec132c5a5617163be3fdcf19dcb"} err="failed to get container status \"4ddba416c5b5733368960574da821a5e7a935ec132c5a5617163be3fdcf19dcb\": rpc error: code = NotFound desc = could not find container \"4ddba416c5b5733368960574da821a5e7a935ec132c5a5617163be3fdcf19dcb\": container with ID starting with 4ddba416c5b5733368960574da821a5e7a935ec132c5a5617163be3fdcf19dcb not found: ID does not exist" Nov 03 20:40:41.431866 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:41.431836 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66c6bd48db-bjx28"] Nov 03 20:40:41.481287 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:41.481252 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-66c6bd48db-bjx28"] Nov 03 20:40:42.892825 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:42.892791 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33077f68-6503-41d4-a597-4f2b403b9340" path="/var/lib/kubelet/pods/33077f68-6503-41d4-a597-4f2b403b9340/volumes" Nov 03 20:40:46.320289 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:40:46.320252 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 20:40:49.486476 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:49.486420 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 20:40:49.486885 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:49.486747 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca" containerName="alertmanager" containerID="cri-o://cf0c1e86c1d12db1ba181f0a4f0df234569f4c716d96569b4dcc8ed9c2a3bdee" gracePeriod=120 Nov 03 20:40:49.486885 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:49.486816 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca" containerName="kube-rbac-proxy-metric" containerID="cri-o://4feeac0fe9271e272a54f5602044c10d9fb124b42ab2eb0c6174df01b15ae09c" gracePeriod=120 Nov 03 20:40:49.487009 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:49.486904 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca" containerName="kube-rbac-proxy" containerID="cri-o://40cb4d8e08c1bd2ac44a28a2720f3b5fc6acb0900185aa0198325e82120dbcc3" gracePeriod=120 Nov 03 20:40:49.487009 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:49.486922 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca" containerName="prom-label-proxy" containerID="cri-o://5858c44f50e2e7cec866c9064cb861cee92f8998b3c4922c3d80bcb7a8793a92" gracePeriod=120 Nov 03 20:40:49.487009 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:49.486999 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca" containerName="kube-rbac-proxy-web" containerID="cri-o://299d01664c3913b3d5da14b18b9d43f1b35e89998919279d8890f9d368166849" gracePeriod=120 Nov 03 20:40:49.487212 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:49.486998 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca" containerName="config-reloader" containerID="cri-o://08214c61e55f9633bf07868bd749eec5e7c72e8d7cd9ade9a4c06659c45a4040" gracePeriod=120 Nov 03 20:40:50.306612 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:50.306579 4305 generic.go:356] "Generic (PLEG): container finished" podID="5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca" containerID="5858c44f50e2e7cec866c9064cb861cee92f8998b3c4922c3d80bcb7a8793a92" exitCode=0 Nov 03 20:40:50.306612 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:50.306603 4305 generic.go:356] "Generic (PLEG): container finished" podID="5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca" containerID="4feeac0fe9271e272a54f5602044c10d9fb124b42ab2eb0c6174df01b15ae09c" exitCode=0 Nov 03 20:40:50.306612 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:50.306610 4305 generic.go:356] "Generic (PLEG): container finished" podID="5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca" containerID="40cb4d8e08c1bd2ac44a28a2720f3b5fc6acb0900185aa0198325e82120dbcc3" exitCode=0 Nov 03 20:40:50.306612 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:50.306617 4305 generic.go:356] "Generic (PLEG): container finished" podID="5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca" containerID="08214c61e55f9633bf07868bd749eec5e7c72e8d7cd9ade9a4c06659c45a4040" exitCode=0 Nov 03 20:40:50.306612 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:50.306624 4305 generic.go:356] "Generic (PLEG): container finished" podID="5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca" containerID="cf0c1e86c1d12db1ba181f0a4f0df234569f4c716d96569b4dcc8ed9c2a3bdee" exitCode=0 Nov 03 20:40:50.306907 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:50.306648 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca","Type":"ContainerDied","Data":"5858c44f50e2e7cec866c9064cb861cee92f8998b3c4922c3d80bcb7a8793a92"} Nov 03 20:40:50.306907 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:50.306713 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca","Type":"ContainerDied","Data":"4feeac0fe9271e272a54f5602044c10d9fb124b42ab2eb0c6174df01b15ae09c"} Nov 03 20:40:50.306907 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:50.306740 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca","Type":"ContainerDied","Data":"40cb4d8e08c1bd2ac44a28a2720f3b5fc6acb0900185aa0198325e82120dbcc3"} Nov 03 20:40:50.306907 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:50.306763 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca","Type":"ContainerDied","Data":"08214c61e55f9633bf07868bd749eec5e7c72e8d7cd9ade9a4c06659c45a4040"} Nov 03 20:40:50.306907 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:50.306787 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca","Type":"ContainerDied","Data":"cf0c1e86c1d12db1ba181f0a4f0df234569f4c716d96569b4dcc8ed9c2a3bdee"} Nov 03 20:40:50.739845 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:50.739819 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:40:50.882961 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:50.882873 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-secret-alertmanager-kube-rbac-proxy\") pod \"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca\" (UID: \"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca\") " Nov 03 20:40:50.882961 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:50.882926 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-web-config\" (UniqueName: \"kubernetes.io/secret/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-web-config\") pod \"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca\" (UID: \"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca\") " Nov 03 20:40:50.882961 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:50.882957 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-secret-alertmanager-kube-rbac-proxy-web\") pod \"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca\" (UID: \"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca\") " Nov 03 20:40:50.883247 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:50.883005 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-secret-alertmanager-kube-rbac-proxy-metric\") pod \"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca\" (UID: \"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca\") " Nov 03 20:40:50.883247 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:50.883045 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-alertmanager-main-db\") pod \"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca\" (UID: \"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca\") " Nov 03 20:40:50.883247 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:50.883076 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-secret-alertmanager-main-tls\") pod \"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca\" (UID: \"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca\") " Nov 03 20:40:50.883247 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:50.883106 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-config-out\" (UniqueName: \"kubernetes.io/empty-dir/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-config-out\") pod \"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca\" (UID: \"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca\") " Nov 03 20:40:50.883247 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:50.883139 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-config-volume\" (UniqueName: \"kubernetes.io/secret/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-config-volume\") pod \"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca\" (UID: \"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca\") " Nov 03 20:40:50.883247 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:50.883181 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-tls-assets\" (UniqueName: \"kubernetes.io/projected/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-tls-assets\") pod \"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca\" (UID: \"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca\") " Nov 03 20:40:50.883247 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:50.883214 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-cluster-tls-config\") pod \"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca\" (UID: \"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca\") " Nov 03 20:40:50.883247 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:50.883248 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-kube-api-access-xbl8b\" (UniqueName: \"kubernetes.io/projected/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-kube-api-access-xbl8b\") pod \"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca\" (UID: \"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca\") " Nov 03 20:40:50.883842 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:50.883317 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-metrics-client-ca\") pod \"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca\" (UID: \"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca\") " Nov 03 20:40:50.883842 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:50.883364 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-alertmanager-trusted-ca-bundle\") pod \"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca\" (UID: \"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca\") " Nov 03 20:40:50.883842 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:50.883676 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-alertmanager-main-db" pod "5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca" (UID: "5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:50.884030 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:50.883998 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-alertmanager-trusted-ca-bundle" pod "5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca" (UID: "5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:40:50.884537 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:50.884246 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-metrics-client-ca" pod "5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca" (UID: "5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:40:50.886351 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:50.886310 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-secret-alertmanager-main-tls" pod "5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca" (UID: "5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:40:50.886642 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:50.886613 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-secret-alertmanager-kube-rbac-proxy" pod "5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca" (UID: "5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:40:50.887093 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:50.886996 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-secret-alertmanager-kube-rbac-proxy-web" pod "5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca" (UID: "5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:40:50.887093 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:50.887058 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-secret-alertmanager-kube-rbac-proxy-metric" pod "5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca" (UID: "5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:40:50.887093 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:50.887063 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-config-out" pod "5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca" (UID: "5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:50.887250 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:50.887192 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-config-volume" pod "5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca" (UID: "5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:40:50.888051 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:50.888030 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-tls-assets" pod "5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca" (UID: "5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:50.888137 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:50.888116 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-kube-api-access-xbl8b" pod "5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca" (UID: "5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca"). InnerVolumeSpecName "kube-api-access-xbl8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:50.891649 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:50.891591 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-cluster-tls-config" pod "5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca" (UID: "5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:40:50.898467 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:50.898423 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-web-config" pod "5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca" (UID: "5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:40:50.984604 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:50.984574 4305 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:40:50.984604 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:50.984602 4305 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:40:50.984819 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:50.984616 4305 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-web-config\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:40:50.984819 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:50.984629 4305 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:40:50.984819 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:50.984643 4305 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:40:50.984819 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:50.984656 4305 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-alertmanager-main-db\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:40:50.984819 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:50.984667 4305 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-secret-alertmanager-main-tls\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:40:50.984819 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:50.984680 4305 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-config-out\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:40:50.984819 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:50.984691 4305 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-config-volume\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:40:50.984819 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:50.984703 4305 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-tls-assets\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:40:50.984819 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:50.984714 4305 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-cluster-tls-config\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:40:50.984819 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:50.984726 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbl8b\" (UniqueName: \"kubernetes.io/projected/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-kube-api-access-xbl8b\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:40:50.984819 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:50.984739 4305 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca-metrics-client-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:40:51.313426 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:51.313392 4305 generic.go:356] "Generic (PLEG): container finished" podID="5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca" containerID="299d01664c3913b3d5da14b18b9d43f1b35e89998919279d8890f9d368166849" exitCode=0 Nov 03 20:40:51.313615 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:51.313479 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca","Type":"ContainerDied","Data":"299d01664c3913b3d5da14b18b9d43f1b35e89998919279d8890f9d368166849"} Nov 03 20:40:51.313615 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:51.313548 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca","Type":"ContainerDied","Data":"718a4392b03fab4a5ce2fb99b0158cf739c31d16ab67c1b6aad9068c083fa0c2"} Nov 03 20:40:51.313615 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:51.313490 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:40:51.313615 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:51.313581 4305 scope.go:122] "RemoveContainer" containerID="5858c44f50e2e7cec866c9064cb861cee92f8998b3c4922c3d80bcb7a8793a92" Nov 03 20:40:51.321591 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:51.321578 4305 scope.go:122] "RemoveContainer" containerID="4feeac0fe9271e272a54f5602044c10d9fb124b42ab2eb0c6174df01b15ae09c" Nov 03 20:40:51.328681 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:51.328666 4305 scope.go:122] "RemoveContainer" containerID="40cb4d8e08c1bd2ac44a28a2720f3b5fc6acb0900185aa0198325e82120dbcc3" Nov 03 20:40:51.335588 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:51.335573 4305 scope.go:122] "RemoveContainer" containerID="299d01664c3913b3d5da14b18b9d43f1b35e89998919279d8890f9d368166849" Nov 03 20:40:51.342348 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:51.342332 4305 scope.go:122] "RemoveContainer" containerID="08214c61e55f9633bf07868bd749eec5e7c72e8d7cd9ade9a4c06659c45a4040" Nov 03 20:40:51.349265 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:51.349250 4305 scope.go:122] "RemoveContainer" containerID="cf0c1e86c1d12db1ba181f0a4f0df234569f4c716d96569b4dcc8ed9c2a3bdee" Nov 03 20:40:51.356025 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:51.356003 4305 scope.go:122] "RemoveContainer" containerID="3ce61a2a2fcd9fa3610ece57f88c54f9661f7e00ce09494bfc2e83db6a6bb6b9" Nov 03 20:40:51.363232 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:51.363213 4305 scope.go:122] "RemoveContainer" containerID="5858c44f50e2e7cec866c9064cb861cee92f8998b3c4922c3d80bcb7a8793a92" Nov 03 20:40:51.363509 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:40:51.363491 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5858c44f50e2e7cec866c9064cb861cee92f8998b3c4922c3d80bcb7a8793a92\": container with ID starting with 5858c44f50e2e7cec866c9064cb861cee92f8998b3c4922c3d80bcb7a8793a92 not found: ID does not exist" containerID="5858c44f50e2e7cec866c9064cb861cee92f8998b3c4922c3d80bcb7a8793a92" Nov 03 20:40:51.363576 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:51.363521 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5858c44f50e2e7cec866c9064cb861cee92f8998b3c4922c3d80bcb7a8793a92"} err="failed to get container status \"5858c44f50e2e7cec866c9064cb861cee92f8998b3c4922c3d80bcb7a8793a92\": rpc error: code = NotFound desc = could not find container \"5858c44f50e2e7cec866c9064cb861cee92f8998b3c4922c3d80bcb7a8793a92\": container with ID starting with 5858c44f50e2e7cec866c9064cb861cee92f8998b3c4922c3d80bcb7a8793a92 not found: ID does not exist" Nov 03 20:40:51.363576 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:51.363539 4305 scope.go:122] "RemoveContainer" containerID="4feeac0fe9271e272a54f5602044c10d9fb124b42ab2eb0c6174df01b15ae09c" Nov 03 20:40:51.363817 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:40:51.363790 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4feeac0fe9271e272a54f5602044c10d9fb124b42ab2eb0c6174df01b15ae09c\": container with ID starting with 4feeac0fe9271e272a54f5602044c10d9fb124b42ab2eb0c6174df01b15ae09c not found: ID does not exist" containerID="4feeac0fe9271e272a54f5602044c10d9fb124b42ab2eb0c6174df01b15ae09c" Nov 03 20:40:51.363862 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:51.363827 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4feeac0fe9271e272a54f5602044c10d9fb124b42ab2eb0c6174df01b15ae09c"} err="failed to get container status \"4feeac0fe9271e272a54f5602044c10d9fb124b42ab2eb0c6174df01b15ae09c\": rpc error: code = NotFound desc = could not find container \"4feeac0fe9271e272a54f5602044c10d9fb124b42ab2eb0c6174df01b15ae09c\": container with ID starting with 4feeac0fe9271e272a54f5602044c10d9fb124b42ab2eb0c6174df01b15ae09c not found: ID does not exist" Nov 03 20:40:51.363862 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:51.363844 4305 scope.go:122] "RemoveContainer" containerID="40cb4d8e08c1bd2ac44a28a2720f3b5fc6acb0900185aa0198325e82120dbcc3" Nov 03 20:40:51.364088 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:40:51.364071 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40cb4d8e08c1bd2ac44a28a2720f3b5fc6acb0900185aa0198325e82120dbcc3\": container with ID starting with 40cb4d8e08c1bd2ac44a28a2720f3b5fc6acb0900185aa0198325e82120dbcc3 not found: ID does not exist" containerID="40cb4d8e08c1bd2ac44a28a2720f3b5fc6acb0900185aa0198325e82120dbcc3" Nov 03 20:40:51.364130 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:51.364097 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40cb4d8e08c1bd2ac44a28a2720f3b5fc6acb0900185aa0198325e82120dbcc3"} err="failed to get container status \"40cb4d8e08c1bd2ac44a28a2720f3b5fc6acb0900185aa0198325e82120dbcc3\": rpc error: code = NotFound desc = could not find container \"40cb4d8e08c1bd2ac44a28a2720f3b5fc6acb0900185aa0198325e82120dbcc3\": container with ID starting with 40cb4d8e08c1bd2ac44a28a2720f3b5fc6acb0900185aa0198325e82120dbcc3 not found: ID does not exist" Nov 03 20:40:51.364130 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:51.364112 4305 scope.go:122] "RemoveContainer" containerID="299d01664c3913b3d5da14b18b9d43f1b35e89998919279d8890f9d368166849" Nov 03 20:40:51.364363 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:40:51.364341 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"299d01664c3913b3d5da14b18b9d43f1b35e89998919279d8890f9d368166849\": container with ID starting with 299d01664c3913b3d5da14b18b9d43f1b35e89998919279d8890f9d368166849 not found: ID does not exist" containerID="299d01664c3913b3d5da14b18b9d43f1b35e89998919279d8890f9d368166849" Nov 03 20:40:51.364423 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:51.364367 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"299d01664c3913b3d5da14b18b9d43f1b35e89998919279d8890f9d368166849"} err="failed to get container status \"299d01664c3913b3d5da14b18b9d43f1b35e89998919279d8890f9d368166849\": rpc error: code = NotFound desc = could not find container \"299d01664c3913b3d5da14b18b9d43f1b35e89998919279d8890f9d368166849\": container with ID starting with 299d01664c3913b3d5da14b18b9d43f1b35e89998919279d8890f9d368166849 not found: ID does not exist" Nov 03 20:40:51.364423 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:51.364382 4305 scope.go:122] "RemoveContainer" containerID="08214c61e55f9633bf07868bd749eec5e7c72e8d7cd9ade9a4c06659c45a4040" Nov 03 20:40:51.364663 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:40:51.364647 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08214c61e55f9633bf07868bd749eec5e7c72e8d7cd9ade9a4c06659c45a4040\": container with ID starting with 08214c61e55f9633bf07868bd749eec5e7c72e8d7cd9ade9a4c06659c45a4040 not found: ID does not exist" containerID="08214c61e55f9633bf07868bd749eec5e7c72e8d7cd9ade9a4c06659c45a4040" Nov 03 20:40:51.364707 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:51.364666 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08214c61e55f9633bf07868bd749eec5e7c72e8d7cd9ade9a4c06659c45a4040"} err="failed to get container status \"08214c61e55f9633bf07868bd749eec5e7c72e8d7cd9ade9a4c06659c45a4040\": rpc error: code = NotFound desc = could not find container \"08214c61e55f9633bf07868bd749eec5e7c72e8d7cd9ade9a4c06659c45a4040\": container with ID starting with 08214c61e55f9633bf07868bd749eec5e7c72e8d7cd9ade9a4c06659c45a4040 not found: ID does not exist" Nov 03 20:40:51.364707 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:51.364679 4305 scope.go:122] "RemoveContainer" containerID="cf0c1e86c1d12db1ba181f0a4f0df234569f4c716d96569b4dcc8ed9c2a3bdee" Nov 03 20:40:51.364880 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:40:51.364864 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf0c1e86c1d12db1ba181f0a4f0df234569f4c716d96569b4dcc8ed9c2a3bdee\": container with ID starting with cf0c1e86c1d12db1ba181f0a4f0df234569f4c716d96569b4dcc8ed9c2a3bdee not found: ID does not exist" containerID="cf0c1e86c1d12db1ba181f0a4f0df234569f4c716d96569b4dcc8ed9c2a3bdee" Nov 03 20:40:51.364937 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:51.364883 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf0c1e86c1d12db1ba181f0a4f0df234569f4c716d96569b4dcc8ed9c2a3bdee"} err="failed to get container status \"cf0c1e86c1d12db1ba181f0a4f0df234569f4c716d96569b4dcc8ed9c2a3bdee\": rpc error: code = NotFound desc = could not find container \"cf0c1e86c1d12db1ba181f0a4f0df234569f4c716d96569b4dcc8ed9c2a3bdee\": container with ID starting with cf0c1e86c1d12db1ba181f0a4f0df234569f4c716d96569b4dcc8ed9c2a3bdee not found: ID does not exist" Nov 03 20:40:51.364937 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:51.364894 4305 scope.go:122] "RemoveContainer" containerID="3ce61a2a2fcd9fa3610ece57f88c54f9661f7e00ce09494bfc2e83db6a6bb6b9" Nov 03 20:40:51.365076 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:40:51.365047 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ce61a2a2fcd9fa3610ece57f88c54f9661f7e00ce09494bfc2e83db6a6bb6b9\": container with ID starting with 3ce61a2a2fcd9fa3610ece57f88c54f9661f7e00ce09494bfc2e83db6a6bb6b9 not found: ID does not exist" containerID="3ce61a2a2fcd9fa3610ece57f88c54f9661f7e00ce09494bfc2e83db6a6bb6b9" Nov 03 20:40:51.365076 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:51.365064 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ce61a2a2fcd9fa3610ece57f88c54f9661f7e00ce09494bfc2e83db6a6bb6b9"} err="failed to get container status \"3ce61a2a2fcd9fa3610ece57f88c54f9661f7e00ce09494bfc2e83db6a6bb6b9\": rpc error: code = NotFound desc = could not find container \"3ce61a2a2fcd9fa3610ece57f88c54f9661f7e00ce09494bfc2e83db6a6bb6b9\": container with ID starting with 3ce61a2a2fcd9fa3610ece57f88c54f9661f7e00ce09494bfc2e83db6a6bb6b9 not found: ID does not exist" Nov 03 20:40:51.479938 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:51.479907 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 20:40:51.529363 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:51.529318 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 20:40:51.912647 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:51.912606 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5b675ff864-tw7z9" podUID="13737861-083d-4ed4-bdb5-2022863b54d4" containerName="console" containerID="cri-o://2851f75a21585a48c3b8b67f4a4291ce7f0390a9833737f3da909f2f6f32255c" gracePeriod=15 Nov 03 20:40:52.160327 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:52.160303 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b675ff864-tw7z9" Nov 03 20:40:52.293321 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:52.293247 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13737861-083d-4ed4-bdb5-2022863b54d4-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/13737861-083d-4ed4-bdb5-2022863b54d4-oauth-serving-cert\") pod \"13737861-083d-4ed4-bdb5-2022863b54d4\" (UID: \"13737861-083d-4ed4-bdb5-2022863b54d4\") " Nov 03 20:40:52.293321 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:52.293290 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13737861-083d-4ed4-bdb5-2022863b54d4-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/13737861-083d-4ed4-bdb5-2022863b54d4-console-serving-cert\") pod \"13737861-083d-4ed4-bdb5-2022863b54d4\" (UID: \"13737861-083d-4ed4-bdb5-2022863b54d4\") " Nov 03 20:40:52.293542 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:52.293342 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13737861-083d-4ed4-bdb5-2022863b54d4-tmp\" (UniqueName: \"kubernetes.io/empty-dir/13737861-083d-4ed4-bdb5-2022863b54d4-tmp\") pod \"13737861-083d-4ed4-bdb5-2022863b54d4\" (UID: \"13737861-083d-4ed4-bdb5-2022863b54d4\") " Nov 03 20:40:52.293542 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:52.293399 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13737861-083d-4ed4-bdb5-2022863b54d4-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/13737861-083d-4ed4-bdb5-2022863b54d4-console-oauth-config\") pod \"13737861-083d-4ed4-bdb5-2022863b54d4\" (UID: \"13737861-083d-4ed4-bdb5-2022863b54d4\") " Nov 03 20:40:52.293542 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:52.293425 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13737861-083d-4ed4-bdb5-2022863b54d4-service-ca\" (UniqueName: \"kubernetes.io/configmap/13737861-083d-4ed4-bdb5-2022863b54d4-service-ca\") pod \"13737861-083d-4ed4-bdb5-2022863b54d4\" (UID: \"13737861-083d-4ed4-bdb5-2022863b54d4\") " Nov 03 20:40:52.293542 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:52.293484 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13737861-083d-4ed4-bdb5-2022863b54d4-kube-api-access-q6b5t\" (UniqueName: \"kubernetes.io/projected/13737861-083d-4ed4-bdb5-2022863b54d4-kube-api-access-q6b5t\") pod \"13737861-083d-4ed4-bdb5-2022863b54d4\" (UID: \"13737861-083d-4ed4-bdb5-2022863b54d4\") " Nov 03 20:40:52.293721 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:52.293602 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13737861-083d-4ed4-bdb5-2022863b54d4-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13737861-083d-4ed4-bdb5-2022863b54d4-trusted-ca-bundle\") pod \"13737861-083d-4ed4-bdb5-2022863b54d4\" (UID: \"13737861-083d-4ed4-bdb5-2022863b54d4\") " Nov 03 20:40:52.293721 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:52.293684 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13737861-083d-4ed4-bdb5-2022863b54d4-console-config\" (UniqueName: \"kubernetes.io/configmap/13737861-083d-4ed4-bdb5-2022863b54d4-console-config\") pod \"13737861-083d-4ed4-bdb5-2022863b54d4\" (UID: \"13737861-083d-4ed4-bdb5-2022863b54d4\") " Nov 03 20:40:52.293721 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:52.293686 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13737861-083d-4ed4-bdb5-2022863b54d4-oauth-serving-cert" pod "13737861-083d-4ed4-bdb5-2022863b54d4" (UID: "13737861-083d-4ed4-bdb5-2022863b54d4"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:40:52.293849 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:52.293821 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13737861-083d-4ed4-bdb5-2022863b54d4-tmp" pod "13737861-083d-4ed4-bdb5-2022863b54d4" (UID: "13737861-083d-4ed4-bdb5-2022863b54d4"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:52.294040 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:52.293981 4305 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/13737861-083d-4ed4-bdb5-2022863b54d4-oauth-serving-cert\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:40:52.294040 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:52.294005 4305 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/13737861-083d-4ed4-bdb5-2022863b54d4-tmp\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:40:52.294199 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:52.294113 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13737861-083d-4ed4-bdb5-2022863b54d4-service-ca" pod "13737861-083d-4ed4-bdb5-2022863b54d4" (UID: "13737861-083d-4ed4-bdb5-2022863b54d4"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:40:52.294199 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:52.294123 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13737861-083d-4ed4-bdb5-2022863b54d4-console-config" pod "13737861-083d-4ed4-bdb5-2022863b54d4" (UID: "13737861-083d-4ed4-bdb5-2022863b54d4"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:40:52.294491 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:52.294279 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13737861-083d-4ed4-bdb5-2022863b54d4-trusted-ca-bundle" pod "13737861-083d-4ed4-bdb5-2022863b54d4" (UID: "13737861-083d-4ed4-bdb5-2022863b54d4"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:40:52.295644 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:52.295619 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13737861-083d-4ed4-bdb5-2022863b54d4-console-serving-cert" pod "13737861-083d-4ed4-bdb5-2022863b54d4" (UID: "13737861-083d-4ed4-bdb5-2022863b54d4"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:40:52.296023 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:52.295990 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13737861-083d-4ed4-bdb5-2022863b54d4-kube-api-access-q6b5t" pod "13737861-083d-4ed4-bdb5-2022863b54d4" (UID: "13737861-083d-4ed4-bdb5-2022863b54d4"). InnerVolumeSpecName "kube-api-access-q6b5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:52.296151 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:52.296033 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13737861-083d-4ed4-bdb5-2022863b54d4-console-oauth-config" pod "13737861-083d-4ed4-bdb5-2022863b54d4" (UID: "13737861-083d-4ed4-bdb5-2022863b54d4"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:40:52.319232 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:52.319203 4305 generic.go:356] "Generic (PLEG): container finished" podID="13737861-083d-4ed4-bdb5-2022863b54d4" containerID="2851f75a21585a48c3b8b67f4a4291ce7f0390a9833737f3da909f2f6f32255c" exitCode=0 Nov 03 20:40:52.319392 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:52.319247 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b675ff864-tw7z9" Nov 03 20:40:52.319392 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:52.319251 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b675ff864-tw7z9" event={"ID":"13737861-083d-4ed4-bdb5-2022863b54d4","Type":"ContainerDied","Data":"2851f75a21585a48c3b8b67f4a4291ce7f0390a9833737f3da909f2f6f32255c"} Nov 03 20:40:52.319392 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:52.319288 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b675ff864-tw7z9" event={"ID":"13737861-083d-4ed4-bdb5-2022863b54d4","Type":"ContainerDied","Data":"878983fd4211468704c61e2946c638f4288b40f56c1658ddbbfc06b13bd9c5f9"} Nov 03 20:40:52.319392 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:52.319306 4305 scope.go:122] "RemoveContainer" containerID="2851f75a21585a48c3b8b67f4a4291ce7f0390a9833737f3da909f2f6f32255c" Nov 03 20:40:52.328116 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:52.328099 4305 scope.go:122] "RemoveContainer" containerID="2851f75a21585a48c3b8b67f4a4291ce7f0390a9833737f3da909f2f6f32255c" Nov 03 20:40:52.328404 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:40:52.328385 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2851f75a21585a48c3b8b67f4a4291ce7f0390a9833737f3da909f2f6f32255c\": container with ID starting with 2851f75a21585a48c3b8b67f4a4291ce7f0390a9833737f3da909f2f6f32255c not found: ID does not exist" containerID="2851f75a21585a48c3b8b67f4a4291ce7f0390a9833737f3da909f2f6f32255c" Nov 03 20:40:52.328487 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:52.328413 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2851f75a21585a48c3b8b67f4a4291ce7f0390a9833737f3da909f2f6f32255c"} err="failed to get container status \"2851f75a21585a48c3b8b67f4a4291ce7f0390a9833737f3da909f2f6f32255c\": rpc error: code = NotFound desc = could not find container \"2851f75a21585a48c3b8b67f4a4291ce7f0390a9833737f3da909f2f6f32255c\": container with ID starting with 2851f75a21585a48c3b8b67f4a4291ce7f0390a9833737f3da909f2f6f32255c not found: ID does not exist" Nov 03 20:40:52.394804 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:52.394769 4305 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/13737861-083d-4ed4-bdb5-2022863b54d4-console-oauth-config\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:40:52.394804 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:52.394796 4305 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/13737861-083d-4ed4-bdb5-2022863b54d4-service-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:40:52.394804 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:52.394807 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6b5t\" (UniqueName: \"kubernetes.io/projected/13737861-083d-4ed4-bdb5-2022863b54d4-kube-api-access-q6b5t\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:40:52.395032 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:52.394817 4305 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13737861-083d-4ed4-bdb5-2022863b54d4-trusted-ca-bundle\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:40:52.395032 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:52.394826 4305 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/13737861-083d-4ed4-bdb5-2022863b54d4-console-config\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:40:52.395032 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:52.394836 4305 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/13737861-083d-4ed4-bdb5-2022863b54d4-console-serving-cert\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:40:52.482671 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:52.482638 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b675ff864-tw7z9"] Nov 03 20:40:52.486746 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:52.486721 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5b675ff864-tw7z9"] Nov 03 20:40:52.892589 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:52.892555 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13737861-083d-4ed4-bdb5-2022863b54d4" path="/var/lib/kubelet/pods/13737861-083d-4ed4-bdb5-2022863b54d4/volumes" Nov 03 20:40:52.892999 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:52.892985 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca" path="/var/lib/kubelet/pods/5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca/volumes" Nov 03 20:40:55.270426 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.270377 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 20:40:55.270960 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.270818 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="34de3dcc-9aa3-4c4a-b827-d8e944c3514c" containerName="prometheus" containerID="cri-o://1c7b0642697af1976a78b786dcf46e1278263bd5d94362657b2908d14d735b59" gracePeriod=600 Nov 03 20:40:55.270960 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.270881 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="34de3dcc-9aa3-4c4a-b827-d8e944c3514c" containerName="thanos-sidecar" containerID="cri-o://0a743ec9b02a2c07e921055a28c26271f7f651cdab2a4120d66794950ebb731e" gracePeriod=600 Nov 03 20:40:55.270960 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.270888 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="34de3dcc-9aa3-4c4a-b827-d8e944c3514c" containerName="kube-rbac-proxy" containerID="cri-o://9924a3acd9fb2dcf053317e355d5e0174fe334929f22a3b18991f251916c34bb" gracePeriod=600 Nov 03 20:40:55.270960 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.270913 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="34de3dcc-9aa3-4c4a-b827-d8e944c3514c" containerName="config-reloader" containerID="cri-o://60ac802a7d17285286485bda07751fb435346c744067be63b6fe15f16d6fd0ef" gracePeriod=600 Nov 03 20:40:55.271167 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.270859 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="34de3dcc-9aa3-4c4a-b827-d8e944c3514c" containerName="kube-rbac-proxy-thanos" containerID="cri-o://a45db29215d088d64aceff9ecab04a73618d50c292f4457f2bc5aa96f78d8ac1" gracePeriod=600 Nov 03 20:40:55.271167 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.271106 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="34de3dcc-9aa3-4c4a-b827-d8e944c3514c" containerName="kube-rbac-proxy-web" containerID="cri-o://d09c16868fd4bc2d1c618abf0d5538842aef98d14056fc9cabe0d320f373ea55" gracePeriod=600 Nov 03 20:40:55.527174 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.527108 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:55.629522 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.629484 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-configmap-metrics-client-ca\") pod \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " Nov 03 20:40:55.629723 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.629542 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-config\" (UniqueName: \"kubernetes.io/secret/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-config\") pod \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " Nov 03 20:40:55.629723 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.629571 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-prometheus-k8s-db\") pod \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " Nov 03 20:40:55.629723 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.629614 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-web-config\" (UniqueName: \"kubernetes.io/secret/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-web-config\") pod \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " Nov 03 20:40:55.629723 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.629646 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-secret-grpc-tls\") pod \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " Nov 03 20:40:55.629723 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.629670 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-prometheus-k8s-rulefiles-0\") pod \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " Nov 03 20:40:55.629723 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.629695 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-config-out\" (UniqueName: \"kubernetes.io/empty-dir/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-config-out\") pod \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " Nov 03 20:40:55.629723 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.629720 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-kube-api-access-h84lh\" (UniqueName: \"kubernetes.io/projected/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-kube-api-access-h84lh\") pod \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " Nov 03 20:40:55.630061 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.629748 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-configmap-serving-certs-ca-bundle\") pod \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " Nov 03 20:40:55.630061 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.629792 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-secret-kube-rbac-proxy\") pod \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " Nov 03 20:40:55.630061 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.629822 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-tls-assets\" (UniqueName: \"kubernetes.io/projected/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-tls-assets\") pod \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " Nov 03 20:40:55.630061 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.629852 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-secret-metrics-client-certs\") pod \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " Nov 03 20:40:55.630061 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.629924 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " Nov 03 20:40:55.630061 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.629919 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-configmap-metrics-client-ca" pod "34de3dcc-9aa3-4c4a-b827-d8e944c3514c" (UID: "34de3dcc-9aa3-4c4a-b827-d8e944c3514c"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:40:55.630061 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.629955 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " Nov 03 20:40:55.630061 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.630010 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-prometheus-k8s-rulefiles-1\") pod \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " Nov 03 20:40:55.630061 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.630039 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-configmap-kubelet-serving-ca-bundle\") pod \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " Nov 03 20:40:55.630501 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.630067 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-prometheus-k8s-rulefiles-2\") pod \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " Nov 03 20:40:55.630501 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.630108 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-secret-prometheus-k8s-tls\") pod \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " Nov 03 20:40:55.630501 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.630135 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-thanos-prometheus-http-client-file\") pod \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " Nov 03 20:40:55.630501 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.630164 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-prometheus-trusted-ca-bundle\") pod \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\" (UID: \"34de3dcc-9aa3-4c4a-b827-d8e944c3514c\") " Nov 03 20:40:55.630501 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.630403 4305 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-configmap-metrics-client-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:40:55.631290 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.630998 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-configmap-kubelet-serving-ca-bundle" pod "34de3dcc-9aa3-4c4a-b827-d8e944c3514c" (UID: "34de3dcc-9aa3-4c4a-b827-d8e944c3514c"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:40:55.631290 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.631258 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-configmap-serving-certs-ca-bundle" pod "34de3dcc-9aa3-4c4a-b827-d8e944c3514c" (UID: "34de3dcc-9aa3-4c4a-b827-d8e944c3514c"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:40:55.632464 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.631601 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-prometheus-trusted-ca-bundle" pod "34de3dcc-9aa3-4c4a-b827-d8e944c3514c" (UID: "34de3dcc-9aa3-4c4a-b827-d8e944c3514c"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:40:55.632464 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.631892 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-prometheus-k8s-rulefiles-1" pod "34de3dcc-9aa3-4c4a-b827-d8e944c3514c" (UID: "34de3dcc-9aa3-4c4a-b827-d8e944c3514c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:40:55.632464 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.632402 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-prometheus-k8s-rulefiles-2" pod "34de3dcc-9aa3-4c4a-b827-d8e944c3514c" (UID: "34de3dcc-9aa3-4c4a-b827-d8e944c3514c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:40:55.633245 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.632654 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-config-out" pod "34de3dcc-9aa3-4c4a-b827-d8e944c3514c" (UID: "34de3dcc-9aa3-4c4a-b827-d8e944c3514c"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:55.633245 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.632690 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-prometheus-k8s-db" pod "34de3dcc-9aa3-4c4a-b827-d8e944c3514c" (UID: "34de3dcc-9aa3-4c4a-b827-d8e944c3514c"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:55.633245 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.633053 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-prometheus-k8s-rulefiles-0" pod "34de3dcc-9aa3-4c4a-b827-d8e944c3514c" (UID: "34de3dcc-9aa3-4c4a-b827-d8e944c3514c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:40:55.633422 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.633241 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-config" pod "34de3dcc-9aa3-4c4a-b827-d8e944c3514c" (UID: "34de3dcc-9aa3-4c4a-b827-d8e944c3514c"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:40:55.633867 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.633839 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-secret-kube-rbac-proxy" pod "34de3dcc-9aa3-4c4a-b827-d8e944c3514c" (UID: "34de3dcc-9aa3-4c4a-b827-d8e944c3514c"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:40:55.634118 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.633892 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-secret-metrics-client-certs" pod "34de3dcc-9aa3-4c4a-b827-d8e944c3514c" (UID: "34de3dcc-9aa3-4c4a-b827-d8e944c3514c"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:40:55.634118 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.633984 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-secret-grpc-tls" pod "34de3dcc-9aa3-4c4a-b827-d8e944c3514c" (UID: "34de3dcc-9aa3-4c4a-b827-d8e944c3514c"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:40:55.634118 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.634044 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-secret-prometheus-k8s-tls" pod "34de3dcc-9aa3-4c4a-b827-d8e944c3514c" (UID: "34de3dcc-9aa3-4c4a-b827-d8e944c3514c"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:40:55.634410 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.634390 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-secret-prometheus-k8s-kube-rbac-proxy-web" pod "34de3dcc-9aa3-4c4a-b827-d8e944c3514c" (UID: "34de3dcc-9aa3-4c4a-b827-d8e944c3514c"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:40:55.635054 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.635037 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-tls-assets" pod "34de3dcc-9aa3-4c4a-b827-d8e944c3514c" (UID: "34de3dcc-9aa3-4c4a-b827-d8e944c3514c"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:55.635326 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.635310 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-thanos-prometheus-http-client-file" pod "34de3dcc-9aa3-4c4a-b827-d8e944c3514c" (UID: "34de3dcc-9aa3-4c4a-b827-d8e944c3514c"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:40:55.635897 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.635876 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-secret-prometheus-k8s-thanos-sidecar-tls" pod "34de3dcc-9aa3-4c4a-b827-d8e944c3514c" (UID: "34de3dcc-9aa3-4c4a-b827-d8e944c3514c"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:40:55.636233 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.636213 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-kube-api-access-h84lh" pod "34de3dcc-9aa3-4c4a-b827-d8e944c3514c" (UID: "34de3dcc-9aa3-4c4a-b827-d8e944c3514c"). InnerVolumeSpecName "kube-api-access-h84lh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:55.644679 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.644655 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-web-config" pod "34de3dcc-9aa3-4c4a-b827-d8e944c3514c" (UID: "34de3dcc-9aa3-4c4a-b827-d8e944c3514c"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:40:55.731269 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.731236 4305 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-config\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:40:55.731269 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.731265 4305 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-prometheus-k8s-db\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:40:55.731269 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.731274 4305 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-web-config\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:40:55.731517 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.731285 4305 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-secret-grpc-tls\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:40:55.731517 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.731293 4305 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:40:55.731517 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.731301 4305 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-config-out\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:40:55.731517 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.731310 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h84lh\" (UniqueName: \"kubernetes.io/projected/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-kube-api-access-h84lh\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:40:55.731517 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.731320 4305 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:40:55.731517 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.731330 4305 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-secret-kube-rbac-proxy\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:40:55.731517 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.731338 4305 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-tls-assets\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:40:55.731517 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.731346 4305 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-secret-metrics-client-certs\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:40:55.731517 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.731355 4305 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:40:55.731517 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.731363 4305 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:40:55.731517 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.731373 4305 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:40:55.731517 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.731382 4305 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:40:55.731517 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.731390 4305 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:40:55.731517 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.731398 4305 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-secret-prometheus-k8s-tls\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:40:55.731517 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.731406 4305 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-thanos-prometheus-http-client-file\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:40:55.731517 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:55.731414 4305 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34de3dcc-9aa3-4c4a-b827-d8e944c3514c-prometheus-trusted-ca-bundle\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:40:56.340278 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.340247 4305 generic.go:356] "Generic (PLEG): container finished" podID="34de3dcc-9aa3-4c4a-b827-d8e944c3514c" containerID="a45db29215d088d64aceff9ecab04a73618d50c292f4457f2bc5aa96f78d8ac1" exitCode=0 Nov 03 20:40:56.340278 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.340273 4305 generic.go:356] "Generic (PLEG): container finished" podID="34de3dcc-9aa3-4c4a-b827-d8e944c3514c" containerID="9924a3acd9fb2dcf053317e355d5e0174fe334929f22a3b18991f251916c34bb" exitCode=0 Nov 03 20:40:56.340278 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.340283 4305 generic.go:356] "Generic (PLEG): container finished" podID="34de3dcc-9aa3-4c4a-b827-d8e944c3514c" containerID="d09c16868fd4bc2d1c618abf0d5538842aef98d14056fc9cabe0d320f373ea55" exitCode=0 Nov 03 20:40:56.340818 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.340292 4305 generic.go:356] "Generic (PLEG): container finished" podID="34de3dcc-9aa3-4c4a-b827-d8e944c3514c" containerID="0a743ec9b02a2c07e921055a28c26271f7f651cdab2a4120d66794950ebb731e" exitCode=0 Nov 03 20:40:56.340818 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.340298 4305 generic.go:356] "Generic (PLEG): container finished" podID="34de3dcc-9aa3-4c4a-b827-d8e944c3514c" containerID="60ac802a7d17285286485bda07751fb435346c744067be63b6fe15f16d6fd0ef" exitCode=0 Nov 03 20:40:56.340818 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.340303 4305 generic.go:356] "Generic (PLEG): container finished" podID="34de3dcc-9aa3-4c4a-b827-d8e944c3514c" containerID="1c7b0642697af1976a78b786dcf46e1278263bd5d94362657b2908d14d735b59" exitCode=0 Nov 03 20:40:56.340818 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.340331 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:56.340818 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.340336 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"34de3dcc-9aa3-4c4a-b827-d8e944c3514c","Type":"ContainerDied","Data":"a45db29215d088d64aceff9ecab04a73618d50c292f4457f2bc5aa96f78d8ac1"} Nov 03 20:40:56.340818 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.340404 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"34de3dcc-9aa3-4c4a-b827-d8e944c3514c","Type":"ContainerDied","Data":"9924a3acd9fb2dcf053317e355d5e0174fe334929f22a3b18991f251916c34bb"} Nov 03 20:40:56.340818 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.340445 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"34de3dcc-9aa3-4c4a-b827-d8e944c3514c","Type":"ContainerDied","Data":"d09c16868fd4bc2d1c618abf0d5538842aef98d14056fc9cabe0d320f373ea55"} Nov 03 20:40:56.340818 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.340460 4305 scope.go:122] "RemoveContainer" containerID="a45db29215d088d64aceff9ecab04a73618d50c292f4457f2bc5aa96f78d8ac1" Nov 03 20:40:56.340818 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.340470 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"34de3dcc-9aa3-4c4a-b827-d8e944c3514c","Type":"ContainerDied","Data":"0a743ec9b02a2c07e921055a28c26271f7f651cdab2a4120d66794950ebb731e"} Nov 03 20:40:56.340818 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.340494 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"34de3dcc-9aa3-4c4a-b827-d8e944c3514c","Type":"ContainerDied","Data":"60ac802a7d17285286485bda07751fb435346c744067be63b6fe15f16d6fd0ef"} Nov 03 20:40:56.340818 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.340516 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"34de3dcc-9aa3-4c4a-b827-d8e944c3514c","Type":"ContainerDied","Data":"1c7b0642697af1976a78b786dcf46e1278263bd5d94362657b2908d14d735b59"} Nov 03 20:40:56.340818 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.340538 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"34de3dcc-9aa3-4c4a-b827-d8e944c3514c","Type":"ContainerDied","Data":"417ede539e89a9735cbab37d491457783b95059daabbf50e1c7c6060c63e16d4"} Nov 03 20:40:56.348381 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.348364 4305 scope.go:122] "RemoveContainer" containerID="9924a3acd9fb2dcf053317e355d5e0174fe334929f22a3b18991f251916c34bb" Nov 03 20:40:56.355261 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.355243 4305 scope.go:122] "RemoveContainer" containerID="d09c16868fd4bc2d1c618abf0d5538842aef98d14056fc9cabe0d320f373ea55" Nov 03 20:40:56.362232 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.362214 4305 scope.go:122] "RemoveContainer" containerID="0a743ec9b02a2c07e921055a28c26271f7f651cdab2a4120d66794950ebb731e" Nov 03 20:40:56.369163 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.369145 4305 scope.go:122] "RemoveContainer" containerID="60ac802a7d17285286485bda07751fb435346c744067be63b6fe15f16d6fd0ef" Nov 03 20:40:56.376377 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.376357 4305 scope.go:122] "RemoveContainer" containerID="1c7b0642697af1976a78b786dcf46e1278263bd5d94362657b2908d14d735b59" Nov 03 20:40:56.383596 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.383574 4305 scope.go:122] "RemoveContainer" containerID="efadbb2d8cc43881c639f3892f83dfb258447b4c5292791e3e4670c1b83a68a8" Nov 03 20:40:56.390122 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.390107 4305 scope.go:122] "RemoveContainer" containerID="a45db29215d088d64aceff9ecab04a73618d50c292f4457f2bc5aa96f78d8ac1" Nov 03 20:40:56.390373 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:40:56.390355 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a45db29215d088d64aceff9ecab04a73618d50c292f4457f2bc5aa96f78d8ac1\": container with ID starting with a45db29215d088d64aceff9ecab04a73618d50c292f4457f2bc5aa96f78d8ac1 not found: ID does not exist" containerID="a45db29215d088d64aceff9ecab04a73618d50c292f4457f2bc5aa96f78d8ac1" Nov 03 20:40:56.390536 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.390384 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a45db29215d088d64aceff9ecab04a73618d50c292f4457f2bc5aa96f78d8ac1"} err="failed to get container status \"a45db29215d088d64aceff9ecab04a73618d50c292f4457f2bc5aa96f78d8ac1\": rpc error: code = NotFound desc = could not find container \"a45db29215d088d64aceff9ecab04a73618d50c292f4457f2bc5aa96f78d8ac1\": container with ID starting with a45db29215d088d64aceff9ecab04a73618d50c292f4457f2bc5aa96f78d8ac1 not found: ID does not exist" Nov 03 20:40:56.390536 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.390402 4305 scope.go:122] "RemoveContainer" containerID="9924a3acd9fb2dcf053317e355d5e0174fe334929f22a3b18991f251916c34bb" Nov 03 20:40:56.390666 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:40:56.390647 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9924a3acd9fb2dcf053317e355d5e0174fe334929f22a3b18991f251916c34bb\": container with ID starting with 9924a3acd9fb2dcf053317e355d5e0174fe334929f22a3b18991f251916c34bb not found: ID does not exist" containerID="9924a3acd9fb2dcf053317e355d5e0174fe334929f22a3b18991f251916c34bb" Nov 03 20:40:56.390704 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.390677 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9924a3acd9fb2dcf053317e355d5e0174fe334929f22a3b18991f251916c34bb"} err="failed to get container status \"9924a3acd9fb2dcf053317e355d5e0174fe334929f22a3b18991f251916c34bb\": rpc error: code = NotFound desc = could not find container \"9924a3acd9fb2dcf053317e355d5e0174fe334929f22a3b18991f251916c34bb\": container with ID starting with 9924a3acd9fb2dcf053317e355d5e0174fe334929f22a3b18991f251916c34bb not found: ID does not exist" Nov 03 20:40:56.390704 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.390695 4305 scope.go:122] "RemoveContainer" containerID="d09c16868fd4bc2d1c618abf0d5538842aef98d14056fc9cabe0d320f373ea55" Nov 03 20:40:56.390932 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:40:56.390915 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d09c16868fd4bc2d1c618abf0d5538842aef98d14056fc9cabe0d320f373ea55\": container with ID starting with d09c16868fd4bc2d1c618abf0d5538842aef98d14056fc9cabe0d320f373ea55 not found: ID does not exist" containerID="d09c16868fd4bc2d1c618abf0d5538842aef98d14056fc9cabe0d320f373ea55" Nov 03 20:40:56.390974 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.390940 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d09c16868fd4bc2d1c618abf0d5538842aef98d14056fc9cabe0d320f373ea55"} err="failed to get container status \"d09c16868fd4bc2d1c618abf0d5538842aef98d14056fc9cabe0d320f373ea55\": rpc error: code = NotFound desc = could not find container \"d09c16868fd4bc2d1c618abf0d5538842aef98d14056fc9cabe0d320f373ea55\": container with ID starting with d09c16868fd4bc2d1c618abf0d5538842aef98d14056fc9cabe0d320f373ea55 not found: ID does not exist" Nov 03 20:40:56.390974 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.390956 4305 scope.go:122] "RemoveContainer" containerID="0a743ec9b02a2c07e921055a28c26271f7f651cdab2a4120d66794950ebb731e" Nov 03 20:40:56.391165 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:40:56.391149 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a743ec9b02a2c07e921055a28c26271f7f651cdab2a4120d66794950ebb731e\": container with ID starting with 0a743ec9b02a2c07e921055a28c26271f7f651cdab2a4120d66794950ebb731e not found: ID does not exist" containerID="0a743ec9b02a2c07e921055a28c26271f7f651cdab2a4120d66794950ebb731e" Nov 03 20:40:56.391201 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.391173 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a743ec9b02a2c07e921055a28c26271f7f651cdab2a4120d66794950ebb731e"} err="failed to get container status \"0a743ec9b02a2c07e921055a28c26271f7f651cdab2a4120d66794950ebb731e\": rpc error: code = NotFound desc = could not find container \"0a743ec9b02a2c07e921055a28c26271f7f651cdab2a4120d66794950ebb731e\": container with ID starting with 0a743ec9b02a2c07e921055a28c26271f7f651cdab2a4120d66794950ebb731e not found: ID does not exist" Nov 03 20:40:56.391201 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.391188 4305 scope.go:122] "RemoveContainer" containerID="60ac802a7d17285286485bda07751fb435346c744067be63b6fe15f16d6fd0ef" Nov 03 20:40:56.391399 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:40:56.391384 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60ac802a7d17285286485bda07751fb435346c744067be63b6fe15f16d6fd0ef\": container with ID starting with 60ac802a7d17285286485bda07751fb435346c744067be63b6fe15f16d6fd0ef not found: ID does not exist" containerID="60ac802a7d17285286485bda07751fb435346c744067be63b6fe15f16d6fd0ef" Nov 03 20:40:56.391504 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.391403 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60ac802a7d17285286485bda07751fb435346c744067be63b6fe15f16d6fd0ef"} err="failed to get container status \"60ac802a7d17285286485bda07751fb435346c744067be63b6fe15f16d6fd0ef\": rpc error: code = NotFound desc = could not find container \"60ac802a7d17285286485bda07751fb435346c744067be63b6fe15f16d6fd0ef\": container with ID starting with 60ac802a7d17285286485bda07751fb435346c744067be63b6fe15f16d6fd0ef not found: ID does not exist" Nov 03 20:40:56.391504 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.391415 4305 scope.go:122] "RemoveContainer" containerID="1c7b0642697af1976a78b786dcf46e1278263bd5d94362657b2908d14d735b59" Nov 03 20:40:56.391910 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:40:56.391895 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c7b0642697af1976a78b786dcf46e1278263bd5d94362657b2908d14d735b59\": container with ID starting with 1c7b0642697af1976a78b786dcf46e1278263bd5d94362657b2908d14d735b59 not found: ID does not exist" containerID="1c7b0642697af1976a78b786dcf46e1278263bd5d94362657b2908d14d735b59" Nov 03 20:40:56.391954 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.391916 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c7b0642697af1976a78b786dcf46e1278263bd5d94362657b2908d14d735b59"} err="failed to get container status \"1c7b0642697af1976a78b786dcf46e1278263bd5d94362657b2908d14d735b59\": rpc error: code = NotFound desc = could not find container \"1c7b0642697af1976a78b786dcf46e1278263bd5d94362657b2908d14d735b59\": container with ID starting with 1c7b0642697af1976a78b786dcf46e1278263bd5d94362657b2908d14d735b59 not found: ID does not exist" Nov 03 20:40:56.391954 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.391930 4305 scope.go:122] "RemoveContainer" containerID="efadbb2d8cc43881c639f3892f83dfb258447b4c5292791e3e4670c1b83a68a8" Nov 03 20:40:56.392190 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:40:56.392175 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efadbb2d8cc43881c639f3892f83dfb258447b4c5292791e3e4670c1b83a68a8\": container with ID starting with efadbb2d8cc43881c639f3892f83dfb258447b4c5292791e3e4670c1b83a68a8 not found: ID does not exist" containerID="efadbb2d8cc43881c639f3892f83dfb258447b4c5292791e3e4670c1b83a68a8" Nov 03 20:40:56.392231 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.392194 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efadbb2d8cc43881c639f3892f83dfb258447b4c5292791e3e4670c1b83a68a8"} err="failed to get container status \"efadbb2d8cc43881c639f3892f83dfb258447b4c5292791e3e4670c1b83a68a8\": rpc error: code = NotFound desc = could not find container \"efadbb2d8cc43881c639f3892f83dfb258447b4c5292791e3e4670c1b83a68a8\": container with ID starting with efadbb2d8cc43881c639f3892f83dfb258447b4c5292791e3e4670c1b83a68a8 not found: ID does not exist" Nov 03 20:40:56.392231 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.392207 4305 scope.go:122] "RemoveContainer" containerID="a45db29215d088d64aceff9ecab04a73618d50c292f4457f2bc5aa96f78d8ac1" Nov 03 20:40:56.392428 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.392411 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a45db29215d088d64aceff9ecab04a73618d50c292f4457f2bc5aa96f78d8ac1"} err="failed to get container status \"a45db29215d088d64aceff9ecab04a73618d50c292f4457f2bc5aa96f78d8ac1\": rpc error: code = NotFound desc = could not find container \"a45db29215d088d64aceff9ecab04a73618d50c292f4457f2bc5aa96f78d8ac1\": container with ID starting with a45db29215d088d64aceff9ecab04a73618d50c292f4457f2bc5aa96f78d8ac1 not found: ID does not exist" Nov 03 20:40:56.392496 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.392448 4305 scope.go:122] "RemoveContainer" containerID="9924a3acd9fb2dcf053317e355d5e0174fe334929f22a3b18991f251916c34bb" Nov 03 20:40:56.392654 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.392637 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9924a3acd9fb2dcf053317e355d5e0174fe334929f22a3b18991f251916c34bb"} err="failed to get container status \"9924a3acd9fb2dcf053317e355d5e0174fe334929f22a3b18991f251916c34bb\": rpc error: code = NotFound desc = could not find container \"9924a3acd9fb2dcf053317e355d5e0174fe334929f22a3b18991f251916c34bb\": container with ID starting with 9924a3acd9fb2dcf053317e355d5e0174fe334929f22a3b18991f251916c34bb not found: ID does not exist" Nov 03 20:40:56.392654 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.392653 4305 scope.go:122] "RemoveContainer" containerID="d09c16868fd4bc2d1c618abf0d5538842aef98d14056fc9cabe0d320f373ea55" Nov 03 20:40:56.392876 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.392858 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d09c16868fd4bc2d1c618abf0d5538842aef98d14056fc9cabe0d320f373ea55"} err="failed to get container status \"d09c16868fd4bc2d1c618abf0d5538842aef98d14056fc9cabe0d320f373ea55\": rpc error: code = NotFound desc = could not find container \"d09c16868fd4bc2d1c618abf0d5538842aef98d14056fc9cabe0d320f373ea55\": container with ID starting with d09c16868fd4bc2d1c618abf0d5538842aef98d14056fc9cabe0d320f373ea55 not found: ID does not exist" Nov 03 20:40:56.392927 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.392876 4305 scope.go:122] "RemoveContainer" containerID="0a743ec9b02a2c07e921055a28c26271f7f651cdab2a4120d66794950ebb731e" Nov 03 20:40:56.393148 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.393128 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a743ec9b02a2c07e921055a28c26271f7f651cdab2a4120d66794950ebb731e"} err="failed to get container status \"0a743ec9b02a2c07e921055a28c26271f7f651cdab2a4120d66794950ebb731e\": rpc error: code = NotFound desc = could not find container \"0a743ec9b02a2c07e921055a28c26271f7f651cdab2a4120d66794950ebb731e\": container with ID starting with 0a743ec9b02a2c07e921055a28c26271f7f651cdab2a4120d66794950ebb731e not found: ID does not exist" Nov 03 20:40:56.393208 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.393156 4305 scope.go:122] "RemoveContainer" containerID="60ac802a7d17285286485bda07751fb435346c744067be63b6fe15f16d6fd0ef" Nov 03 20:40:56.393391 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.393372 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60ac802a7d17285286485bda07751fb435346c744067be63b6fe15f16d6fd0ef"} err="failed to get container status \"60ac802a7d17285286485bda07751fb435346c744067be63b6fe15f16d6fd0ef\": rpc error: code = NotFound desc = could not find container \"60ac802a7d17285286485bda07751fb435346c744067be63b6fe15f16d6fd0ef\": container with ID starting with 60ac802a7d17285286485bda07751fb435346c744067be63b6fe15f16d6fd0ef not found: ID does not exist" Nov 03 20:40:56.393460 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.393393 4305 scope.go:122] "RemoveContainer" containerID="1c7b0642697af1976a78b786dcf46e1278263bd5d94362657b2908d14d735b59" Nov 03 20:40:56.393639 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.393619 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c7b0642697af1976a78b786dcf46e1278263bd5d94362657b2908d14d735b59"} err="failed to get container status \"1c7b0642697af1976a78b786dcf46e1278263bd5d94362657b2908d14d735b59\": rpc error: code = NotFound desc = could not find container \"1c7b0642697af1976a78b786dcf46e1278263bd5d94362657b2908d14d735b59\": container with ID starting with 1c7b0642697af1976a78b786dcf46e1278263bd5d94362657b2908d14d735b59 not found: ID does not exist" Nov 03 20:40:56.393713 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.393642 4305 scope.go:122] "RemoveContainer" containerID="efadbb2d8cc43881c639f3892f83dfb258447b4c5292791e3e4670c1b83a68a8" Nov 03 20:40:56.393879 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.393862 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efadbb2d8cc43881c639f3892f83dfb258447b4c5292791e3e4670c1b83a68a8"} err="failed to get container status \"efadbb2d8cc43881c639f3892f83dfb258447b4c5292791e3e4670c1b83a68a8\": rpc error: code = NotFound desc = could not find container \"efadbb2d8cc43881c639f3892f83dfb258447b4c5292791e3e4670c1b83a68a8\": container with ID starting with efadbb2d8cc43881c639f3892f83dfb258447b4c5292791e3e4670c1b83a68a8 not found: ID does not exist" Nov 03 20:40:56.393931 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.393880 4305 scope.go:122] "RemoveContainer" containerID="a45db29215d088d64aceff9ecab04a73618d50c292f4457f2bc5aa96f78d8ac1" Nov 03 20:40:56.394103 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.394084 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a45db29215d088d64aceff9ecab04a73618d50c292f4457f2bc5aa96f78d8ac1"} err="failed to get container status \"a45db29215d088d64aceff9ecab04a73618d50c292f4457f2bc5aa96f78d8ac1\": rpc error: code = NotFound desc = could not find container \"a45db29215d088d64aceff9ecab04a73618d50c292f4457f2bc5aa96f78d8ac1\": container with ID starting with a45db29215d088d64aceff9ecab04a73618d50c292f4457f2bc5aa96f78d8ac1 not found: ID does not exist" Nov 03 20:40:56.394174 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.394105 4305 scope.go:122] "RemoveContainer" containerID="9924a3acd9fb2dcf053317e355d5e0174fe334929f22a3b18991f251916c34bb" Nov 03 20:40:56.394351 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.394334 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9924a3acd9fb2dcf053317e355d5e0174fe334929f22a3b18991f251916c34bb"} err="failed to get container status \"9924a3acd9fb2dcf053317e355d5e0174fe334929f22a3b18991f251916c34bb\": rpc error: code = NotFound desc = could not find container \"9924a3acd9fb2dcf053317e355d5e0174fe334929f22a3b18991f251916c34bb\": container with ID starting with 9924a3acd9fb2dcf053317e355d5e0174fe334929f22a3b18991f251916c34bb not found: ID does not exist" Nov 03 20:40:56.394422 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.394355 4305 scope.go:122] "RemoveContainer" containerID="d09c16868fd4bc2d1c618abf0d5538842aef98d14056fc9cabe0d320f373ea55" Nov 03 20:40:56.394614 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.394591 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d09c16868fd4bc2d1c618abf0d5538842aef98d14056fc9cabe0d320f373ea55"} err="failed to get container status \"d09c16868fd4bc2d1c618abf0d5538842aef98d14056fc9cabe0d320f373ea55\": rpc error: code = NotFound desc = could not find container \"d09c16868fd4bc2d1c618abf0d5538842aef98d14056fc9cabe0d320f373ea55\": container with ID starting with d09c16868fd4bc2d1c618abf0d5538842aef98d14056fc9cabe0d320f373ea55 not found: ID does not exist" Nov 03 20:40:56.394614 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.394609 4305 scope.go:122] "RemoveContainer" containerID="0a743ec9b02a2c07e921055a28c26271f7f651cdab2a4120d66794950ebb731e" Nov 03 20:40:56.394830 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.394813 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a743ec9b02a2c07e921055a28c26271f7f651cdab2a4120d66794950ebb731e"} err="failed to get container status \"0a743ec9b02a2c07e921055a28c26271f7f651cdab2a4120d66794950ebb731e\": rpc error: code = NotFound desc = could not find container \"0a743ec9b02a2c07e921055a28c26271f7f651cdab2a4120d66794950ebb731e\": container with ID starting with 0a743ec9b02a2c07e921055a28c26271f7f651cdab2a4120d66794950ebb731e not found: ID does not exist" Nov 03 20:40:56.394875 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.394831 4305 scope.go:122] "RemoveContainer" containerID="60ac802a7d17285286485bda07751fb435346c744067be63b6fe15f16d6fd0ef" Nov 03 20:40:56.395031 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.395015 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60ac802a7d17285286485bda07751fb435346c744067be63b6fe15f16d6fd0ef"} err="failed to get container status \"60ac802a7d17285286485bda07751fb435346c744067be63b6fe15f16d6fd0ef\": rpc error: code = NotFound desc = could not find container \"60ac802a7d17285286485bda07751fb435346c744067be63b6fe15f16d6fd0ef\": container with ID starting with 60ac802a7d17285286485bda07751fb435346c744067be63b6fe15f16d6fd0ef not found: ID does not exist" Nov 03 20:40:56.395072 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.395031 4305 scope.go:122] "RemoveContainer" containerID="1c7b0642697af1976a78b786dcf46e1278263bd5d94362657b2908d14d735b59" Nov 03 20:40:56.395242 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.395224 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c7b0642697af1976a78b786dcf46e1278263bd5d94362657b2908d14d735b59"} err="failed to get container status \"1c7b0642697af1976a78b786dcf46e1278263bd5d94362657b2908d14d735b59\": rpc error: code = NotFound desc = could not find container \"1c7b0642697af1976a78b786dcf46e1278263bd5d94362657b2908d14d735b59\": container with ID starting with 1c7b0642697af1976a78b786dcf46e1278263bd5d94362657b2908d14d735b59 not found: ID does not exist" Nov 03 20:40:56.395307 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.395244 4305 scope.go:122] "RemoveContainer" containerID="efadbb2d8cc43881c639f3892f83dfb258447b4c5292791e3e4670c1b83a68a8" Nov 03 20:40:56.395470 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.395448 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efadbb2d8cc43881c639f3892f83dfb258447b4c5292791e3e4670c1b83a68a8"} err="failed to get container status \"efadbb2d8cc43881c639f3892f83dfb258447b4c5292791e3e4670c1b83a68a8\": rpc error: code = NotFound desc = could not find container \"efadbb2d8cc43881c639f3892f83dfb258447b4c5292791e3e4670c1b83a68a8\": container with ID starting with efadbb2d8cc43881c639f3892f83dfb258447b4c5292791e3e4670c1b83a68a8 not found: ID does not exist" Nov 03 20:40:56.395470 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.395468 4305 scope.go:122] "RemoveContainer" containerID="a45db29215d088d64aceff9ecab04a73618d50c292f4457f2bc5aa96f78d8ac1" Nov 03 20:40:56.395711 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.395695 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a45db29215d088d64aceff9ecab04a73618d50c292f4457f2bc5aa96f78d8ac1"} err="failed to get container status \"a45db29215d088d64aceff9ecab04a73618d50c292f4457f2bc5aa96f78d8ac1\": rpc error: code = NotFound desc = could not find container \"a45db29215d088d64aceff9ecab04a73618d50c292f4457f2bc5aa96f78d8ac1\": container with ID starting with a45db29215d088d64aceff9ecab04a73618d50c292f4457f2bc5aa96f78d8ac1 not found: ID does not exist" Nov 03 20:40:56.395711 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.395711 4305 scope.go:122] "RemoveContainer" containerID="9924a3acd9fb2dcf053317e355d5e0174fe334929f22a3b18991f251916c34bb" Nov 03 20:40:56.395939 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.395923 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9924a3acd9fb2dcf053317e355d5e0174fe334929f22a3b18991f251916c34bb"} err="failed to get container status \"9924a3acd9fb2dcf053317e355d5e0174fe334929f22a3b18991f251916c34bb\": rpc error: code = NotFound desc = could not find container \"9924a3acd9fb2dcf053317e355d5e0174fe334929f22a3b18991f251916c34bb\": container with ID starting with 9924a3acd9fb2dcf053317e355d5e0174fe334929f22a3b18991f251916c34bb not found: ID does not exist" Nov 03 20:40:56.395980 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.395939 4305 scope.go:122] "RemoveContainer" containerID="d09c16868fd4bc2d1c618abf0d5538842aef98d14056fc9cabe0d320f373ea55" Nov 03 20:40:56.396162 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.396147 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d09c16868fd4bc2d1c618abf0d5538842aef98d14056fc9cabe0d320f373ea55"} err="failed to get container status \"d09c16868fd4bc2d1c618abf0d5538842aef98d14056fc9cabe0d320f373ea55\": rpc error: code = NotFound desc = could not find container \"d09c16868fd4bc2d1c618abf0d5538842aef98d14056fc9cabe0d320f373ea55\": container with ID starting with d09c16868fd4bc2d1c618abf0d5538842aef98d14056fc9cabe0d320f373ea55 not found: ID does not exist" Nov 03 20:40:56.396162 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.396161 4305 scope.go:122] "RemoveContainer" containerID="0a743ec9b02a2c07e921055a28c26271f7f651cdab2a4120d66794950ebb731e" Nov 03 20:40:56.396368 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.396349 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a743ec9b02a2c07e921055a28c26271f7f651cdab2a4120d66794950ebb731e"} err="failed to get container status \"0a743ec9b02a2c07e921055a28c26271f7f651cdab2a4120d66794950ebb731e\": rpc error: code = NotFound desc = could not find container \"0a743ec9b02a2c07e921055a28c26271f7f651cdab2a4120d66794950ebb731e\": container with ID starting with 0a743ec9b02a2c07e921055a28c26271f7f651cdab2a4120d66794950ebb731e not found: ID does not exist" Nov 03 20:40:56.396418 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.396369 4305 scope.go:122] "RemoveContainer" containerID="60ac802a7d17285286485bda07751fb435346c744067be63b6fe15f16d6fd0ef" Nov 03 20:40:56.396608 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.396583 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60ac802a7d17285286485bda07751fb435346c744067be63b6fe15f16d6fd0ef"} err="failed to get container status \"60ac802a7d17285286485bda07751fb435346c744067be63b6fe15f16d6fd0ef\": rpc error: code = NotFound desc = could not find container \"60ac802a7d17285286485bda07751fb435346c744067be63b6fe15f16d6fd0ef\": container with ID starting with 60ac802a7d17285286485bda07751fb435346c744067be63b6fe15f16d6fd0ef not found: ID does not exist" Nov 03 20:40:56.396608 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.396608 4305 scope.go:122] "RemoveContainer" containerID="1c7b0642697af1976a78b786dcf46e1278263bd5d94362657b2908d14d735b59" Nov 03 20:40:56.396824 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.396797 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c7b0642697af1976a78b786dcf46e1278263bd5d94362657b2908d14d735b59"} err="failed to get container status \"1c7b0642697af1976a78b786dcf46e1278263bd5d94362657b2908d14d735b59\": rpc error: code = NotFound desc = could not find container \"1c7b0642697af1976a78b786dcf46e1278263bd5d94362657b2908d14d735b59\": container with ID starting with 1c7b0642697af1976a78b786dcf46e1278263bd5d94362657b2908d14d735b59 not found: ID does not exist" Nov 03 20:40:56.396824 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.396815 4305 scope.go:122] "RemoveContainer" containerID="efadbb2d8cc43881c639f3892f83dfb258447b4c5292791e3e4670c1b83a68a8" Nov 03 20:40:56.397043 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.397028 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efadbb2d8cc43881c639f3892f83dfb258447b4c5292791e3e4670c1b83a68a8"} err="failed to get container status \"efadbb2d8cc43881c639f3892f83dfb258447b4c5292791e3e4670c1b83a68a8\": rpc error: code = NotFound desc = could not find container \"efadbb2d8cc43881c639f3892f83dfb258447b4c5292791e3e4670c1b83a68a8\": container with ID starting with efadbb2d8cc43881c639f3892f83dfb258447b4c5292791e3e4670c1b83a68a8 not found: ID does not exist" Nov 03 20:40:56.397097 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.397043 4305 scope.go:122] "RemoveContainer" containerID="a45db29215d088d64aceff9ecab04a73618d50c292f4457f2bc5aa96f78d8ac1" Nov 03 20:40:56.397298 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.397282 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a45db29215d088d64aceff9ecab04a73618d50c292f4457f2bc5aa96f78d8ac1"} err="failed to get container status \"a45db29215d088d64aceff9ecab04a73618d50c292f4457f2bc5aa96f78d8ac1\": rpc error: code = NotFound desc = could not find container \"a45db29215d088d64aceff9ecab04a73618d50c292f4457f2bc5aa96f78d8ac1\": container with ID starting with a45db29215d088d64aceff9ecab04a73618d50c292f4457f2bc5aa96f78d8ac1 not found: ID does not exist" Nov 03 20:40:56.397356 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.397299 4305 scope.go:122] "RemoveContainer" containerID="9924a3acd9fb2dcf053317e355d5e0174fe334929f22a3b18991f251916c34bb" Nov 03 20:40:56.397528 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.397511 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9924a3acd9fb2dcf053317e355d5e0174fe334929f22a3b18991f251916c34bb"} err="failed to get container status \"9924a3acd9fb2dcf053317e355d5e0174fe334929f22a3b18991f251916c34bb\": rpc error: code = NotFound desc = could not find container \"9924a3acd9fb2dcf053317e355d5e0174fe334929f22a3b18991f251916c34bb\": container with ID starting with 9924a3acd9fb2dcf053317e355d5e0174fe334929f22a3b18991f251916c34bb not found: ID does not exist" Nov 03 20:40:56.397575 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.397528 4305 scope.go:122] "RemoveContainer" containerID="d09c16868fd4bc2d1c618abf0d5538842aef98d14056fc9cabe0d320f373ea55" Nov 03 20:40:56.397703 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.397683 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d09c16868fd4bc2d1c618abf0d5538842aef98d14056fc9cabe0d320f373ea55"} err="failed to get container status \"d09c16868fd4bc2d1c618abf0d5538842aef98d14056fc9cabe0d320f373ea55\": rpc error: code = NotFound desc = could not find container \"d09c16868fd4bc2d1c618abf0d5538842aef98d14056fc9cabe0d320f373ea55\": container with ID starting with d09c16868fd4bc2d1c618abf0d5538842aef98d14056fc9cabe0d320f373ea55 not found: ID does not exist" Nov 03 20:40:56.397758 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.397703 4305 scope.go:122] "RemoveContainer" containerID="0a743ec9b02a2c07e921055a28c26271f7f651cdab2a4120d66794950ebb731e" Nov 03 20:40:56.397917 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.397884 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a743ec9b02a2c07e921055a28c26271f7f651cdab2a4120d66794950ebb731e"} err="failed to get container status \"0a743ec9b02a2c07e921055a28c26271f7f651cdab2a4120d66794950ebb731e\": rpc error: code = NotFound desc = could not find container \"0a743ec9b02a2c07e921055a28c26271f7f651cdab2a4120d66794950ebb731e\": container with ID starting with 0a743ec9b02a2c07e921055a28c26271f7f651cdab2a4120d66794950ebb731e not found: ID does not exist" Nov 03 20:40:56.397965 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.397917 4305 scope.go:122] "RemoveContainer" containerID="60ac802a7d17285286485bda07751fb435346c744067be63b6fe15f16d6fd0ef" Nov 03 20:40:56.398113 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.398097 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60ac802a7d17285286485bda07751fb435346c744067be63b6fe15f16d6fd0ef"} err="failed to get container status \"60ac802a7d17285286485bda07751fb435346c744067be63b6fe15f16d6fd0ef\": rpc error: code = NotFound desc = could not find container \"60ac802a7d17285286485bda07751fb435346c744067be63b6fe15f16d6fd0ef\": container with ID starting with 60ac802a7d17285286485bda07751fb435346c744067be63b6fe15f16d6fd0ef not found: ID does not exist" Nov 03 20:40:56.398155 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.398114 4305 scope.go:122] "RemoveContainer" containerID="1c7b0642697af1976a78b786dcf46e1278263bd5d94362657b2908d14d735b59" Nov 03 20:40:56.402158 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.399680 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c7b0642697af1976a78b786dcf46e1278263bd5d94362657b2908d14d735b59"} err="failed to get container status \"1c7b0642697af1976a78b786dcf46e1278263bd5d94362657b2908d14d735b59\": rpc error: code = NotFound desc = could not find container \"1c7b0642697af1976a78b786dcf46e1278263bd5d94362657b2908d14d735b59\": container with ID starting with 1c7b0642697af1976a78b786dcf46e1278263bd5d94362657b2908d14d735b59 not found: ID does not exist" Nov 03 20:40:56.402158 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.399709 4305 scope.go:122] "RemoveContainer" containerID="efadbb2d8cc43881c639f3892f83dfb258447b4c5292791e3e4670c1b83a68a8" Nov 03 20:40:56.402426 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.402403 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efadbb2d8cc43881c639f3892f83dfb258447b4c5292791e3e4670c1b83a68a8"} err="failed to get container status \"efadbb2d8cc43881c639f3892f83dfb258447b4c5292791e3e4670c1b83a68a8\": rpc error: code = NotFound desc = could not find container \"efadbb2d8cc43881c639f3892f83dfb258447b4c5292791e3e4670c1b83a68a8\": container with ID starting with efadbb2d8cc43881c639f3892f83dfb258447b4c5292791e3e4670c1b83a68a8 not found: ID does not exist" Nov 03 20:40:56.402551 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.402427 4305 scope.go:122] "RemoveContainer" containerID="a45db29215d088d64aceff9ecab04a73618d50c292f4457f2bc5aa96f78d8ac1" Nov 03 20:40:56.403105 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.403078 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a45db29215d088d64aceff9ecab04a73618d50c292f4457f2bc5aa96f78d8ac1"} err="failed to get container status \"a45db29215d088d64aceff9ecab04a73618d50c292f4457f2bc5aa96f78d8ac1\": rpc error: code = NotFound desc = could not find container \"a45db29215d088d64aceff9ecab04a73618d50c292f4457f2bc5aa96f78d8ac1\": container with ID starting with a45db29215d088d64aceff9ecab04a73618d50c292f4457f2bc5aa96f78d8ac1 not found: ID does not exist" Nov 03 20:40:56.403180 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.403106 4305 scope.go:122] "RemoveContainer" containerID="9924a3acd9fb2dcf053317e355d5e0174fe334929f22a3b18991f251916c34bb" Nov 03 20:40:56.403360 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.403343 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9924a3acd9fb2dcf053317e355d5e0174fe334929f22a3b18991f251916c34bb"} err="failed to get container status \"9924a3acd9fb2dcf053317e355d5e0174fe334929f22a3b18991f251916c34bb\": rpc error: code = NotFound desc = could not find container \"9924a3acd9fb2dcf053317e355d5e0174fe334929f22a3b18991f251916c34bb\": container with ID starting with 9924a3acd9fb2dcf053317e355d5e0174fe334929f22a3b18991f251916c34bb not found: ID does not exist" Nov 03 20:40:56.403406 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.403361 4305 scope.go:122] "RemoveContainer" containerID="d09c16868fd4bc2d1c618abf0d5538842aef98d14056fc9cabe0d320f373ea55" Nov 03 20:40:56.403616 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.403596 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d09c16868fd4bc2d1c618abf0d5538842aef98d14056fc9cabe0d320f373ea55"} err="failed to get container status \"d09c16868fd4bc2d1c618abf0d5538842aef98d14056fc9cabe0d320f373ea55\": rpc error: code = NotFound desc = could not find container \"d09c16868fd4bc2d1c618abf0d5538842aef98d14056fc9cabe0d320f373ea55\": container with ID starting with d09c16868fd4bc2d1c618abf0d5538842aef98d14056fc9cabe0d320f373ea55 not found: ID does not exist" Nov 03 20:40:56.403668 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.403617 4305 scope.go:122] "RemoveContainer" containerID="0a743ec9b02a2c07e921055a28c26271f7f651cdab2a4120d66794950ebb731e" Nov 03 20:40:56.403858 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.403838 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a743ec9b02a2c07e921055a28c26271f7f651cdab2a4120d66794950ebb731e"} err="failed to get container status \"0a743ec9b02a2c07e921055a28c26271f7f651cdab2a4120d66794950ebb731e\": rpc error: code = NotFound desc = could not find container \"0a743ec9b02a2c07e921055a28c26271f7f651cdab2a4120d66794950ebb731e\": container with ID starting with 0a743ec9b02a2c07e921055a28c26271f7f651cdab2a4120d66794950ebb731e not found: ID does not exist" Nov 03 20:40:56.403858 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.403857 4305 scope.go:122] "RemoveContainer" containerID="60ac802a7d17285286485bda07751fb435346c744067be63b6fe15f16d6fd0ef" Nov 03 20:40:56.404096 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.404074 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60ac802a7d17285286485bda07751fb435346c744067be63b6fe15f16d6fd0ef"} err="failed to get container status \"60ac802a7d17285286485bda07751fb435346c744067be63b6fe15f16d6fd0ef\": rpc error: code = NotFound desc = could not find container \"60ac802a7d17285286485bda07751fb435346c744067be63b6fe15f16d6fd0ef\": container with ID starting with 60ac802a7d17285286485bda07751fb435346c744067be63b6fe15f16d6fd0ef not found: ID does not exist" Nov 03 20:40:56.404166 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.404097 4305 scope.go:122] "RemoveContainer" containerID="1c7b0642697af1976a78b786dcf46e1278263bd5d94362657b2908d14d735b59" Nov 03 20:40:56.404295 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.404273 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c7b0642697af1976a78b786dcf46e1278263bd5d94362657b2908d14d735b59"} err="failed to get container status \"1c7b0642697af1976a78b786dcf46e1278263bd5d94362657b2908d14d735b59\": rpc error: code = NotFound desc = could not find container \"1c7b0642697af1976a78b786dcf46e1278263bd5d94362657b2908d14d735b59\": container with ID starting with 1c7b0642697af1976a78b786dcf46e1278263bd5d94362657b2908d14d735b59 not found: ID does not exist" Nov 03 20:40:56.404295 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.404293 4305 scope.go:122] "RemoveContainer" containerID="efadbb2d8cc43881c639f3892f83dfb258447b4c5292791e3e4670c1b83a68a8" Nov 03 20:40:56.404554 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.404539 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efadbb2d8cc43881c639f3892f83dfb258447b4c5292791e3e4670c1b83a68a8"} err="failed to get container status \"efadbb2d8cc43881c639f3892f83dfb258447b4c5292791e3e4670c1b83a68a8\": rpc error: code = NotFound desc = could not find container \"efadbb2d8cc43881c639f3892f83dfb258447b4c5292791e3e4670c1b83a68a8\": container with ID starting with efadbb2d8cc43881c639f3892f83dfb258447b4c5292791e3e4670c1b83a68a8 not found: ID does not exist" Nov 03 20:40:56.507558 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.507529 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 20:40:56.557368 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.557278 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 20:40:56.558201 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.558169 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 20:40:56.558528 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.558506 4305 cpu_manager.go:414] "containerMap: removing container" podUID="5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca" containerName="kube-rbac-proxy-web" Nov 03 20:40:56.558528 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.558524 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca" containerName="kube-rbac-proxy-web" Nov 03 20:40:56.558728 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.558544 4305 cpu_manager.go:414] "containerMap: removing container" podUID="5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca" containerName="config-reloader" Nov 03 20:40:56.558728 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.558551 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca" containerName="config-reloader" Nov 03 20:40:56.558728 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.558563 4305 cpu_manager.go:414] "containerMap: removing container" podUID="33077f68-6503-41d4-a597-4f2b403b9340" containerName="console" Nov 03 20:40:56.558728 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.558569 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33077f68-6503-41d4-a597-4f2b403b9340" containerName="console" Nov 03 20:40:56.558728 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.558578 4305 cpu_manager.go:414] "containerMap: removing container" podUID="34de3dcc-9aa3-4c4a-b827-d8e944c3514c" containerName="kube-rbac-proxy-web" Nov 03 20:40:56.558728 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.558585 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34de3dcc-9aa3-4c4a-b827-d8e944c3514c" containerName="kube-rbac-proxy-web" Nov 03 20:40:56.558728 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.558598 4305 cpu_manager.go:414] "containerMap: removing container" podUID="5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca" containerName="prom-label-proxy" Nov 03 20:40:56.558728 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.558605 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca" containerName="prom-label-proxy" Nov 03 20:40:56.558728 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.558615 4305 cpu_manager.go:414] "containerMap: removing container" podUID="34de3dcc-9aa3-4c4a-b827-d8e944c3514c" containerName="kube-rbac-proxy-thanos" Nov 03 20:40:56.558728 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.558622 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34de3dcc-9aa3-4c4a-b827-d8e944c3514c" containerName="kube-rbac-proxy-thanos" Nov 03 20:40:56.558728 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.558637 4305 cpu_manager.go:414] "containerMap: removing container" podUID="5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca" containerName="kube-rbac-proxy" Nov 03 20:40:56.558728 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.558644 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca" containerName="kube-rbac-proxy" Nov 03 20:40:56.558728 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.558652 4305 cpu_manager.go:414] "containerMap: removing container" podUID="34de3dcc-9aa3-4c4a-b827-d8e944c3514c" containerName="init-config-reloader" Nov 03 20:40:56.558728 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.558659 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34de3dcc-9aa3-4c4a-b827-d8e944c3514c" containerName="init-config-reloader" Nov 03 20:40:56.558728 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.558669 4305 cpu_manager.go:414] "containerMap: removing container" podUID="34de3dcc-9aa3-4c4a-b827-d8e944c3514c" containerName="config-reloader" Nov 03 20:40:56.558728 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.558675 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34de3dcc-9aa3-4c4a-b827-d8e944c3514c" containerName="config-reloader" Nov 03 20:40:56.558728 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.558685 4305 cpu_manager.go:414] "containerMap: removing container" podUID="34de3dcc-9aa3-4c4a-b827-d8e944c3514c" containerName="thanos-sidecar" Nov 03 20:40:56.558728 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.558691 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34de3dcc-9aa3-4c4a-b827-d8e944c3514c" containerName="thanos-sidecar" Nov 03 20:40:56.558728 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.558700 4305 cpu_manager.go:414] "containerMap: removing container" podUID="34de3dcc-9aa3-4c4a-b827-d8e944c3514c" containerName="kube-rbac-proxy" Nov 03 20:40:56.558728 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.558707 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34de3dcc-9aa3-4c4a-b827-d8e944c3514c" containerName="kube-rbac-proxy" Nov 03 20:40:56.558728 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.558719 4305 cpu_manager.go:414] "containerMap: removing container" podUID="5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca" containerName="init-config-reloader" Nov 03 20:40:56.558728 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.558726 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca" containerName="init-config-reloader" Nov 03 20:40:56.558728 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.558735 4305 cpu_manager.go:414] "containerMap: removing container" podUID="5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca" containerName="kube-rbac-proxy-metric" Nov 03 20:40:56.559781 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.558741 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca" containerName="kube-rbac-proxy-metric" Nov 03 20:40:56.559781 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.558753 4305 cpu_manager.go:414] "containerMap: removing container" podUID="13737861-083d-4ed4-bdb5-2022863b54d4" containerName="console" Nov 03 20:40:56.559781 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.558760 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13737861-083d-4ed4-bdb5-2022863b54d4" containerName="console" Nov 03 20:40:56.559781 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.558769 4305 cpu_manager.go:414] "containerMap: removing container" podUID="5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca" containerName="alertmanager" Nov 03 20:40:56.559781 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.558776 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca" containerName="alertmanager" Nov 03 20:40:56.559781 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.558787 4305 cpu_manager.go:414] "containerMap: removing container" podUID="34de3dcc-9aa3-4c4a-b827-d8e944c3514c" containerName="prometheus" Nov 03 20:40:56.559781 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.558794 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34de3dcc-9aa3-4c4a-b827-d8e944c3514c" containerName="prometheus" Nov 03 20:40:56.559781 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.558872 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca" containerName="kube-rbac-proxy-metric" Nov 03 20:40:56.559781 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.558884 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="13737861-083d-4ed4-bdb5-2022863b54d4" containerName="console" Nov 03 20:40:56.559781 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.558893 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca" containerName="alertmanager" Nov 03 20:40:56.559781 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.558901 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="34de3dcc-9aa3-4c4a-b827-d8e944c3514c" containerName="prometheus" Nov 03 20:40:56.559781 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.558912 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca" containerName="kube-rbac-proxy-web" Nov 03 20:40:56.559781 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.558921 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca" containerName="config-reloader" Nov 03 20:40:56.559781 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.558929 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="33077f68-6503-41d4-a597-4f2b403b9340" containerName="console" Nov 03 20:40:56.559781 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.558937 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="34de3dcc-9aa3-4c4a-b827-d8e944c3514c" containerName="kube-rbac-proxy-web" Nov 03 20:40:56.559781 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.558946 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca" containerName="prom-label-proxy" Nov 03 20:40:56.559781 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.558954 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="34de3dcc-9aa3-4c4a-b827-d8e944c3514c" containerName="kube-rbac-proxy-thanos" Nov 03 20:40:56.559781 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.558961 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b1935d2-f61f-4c0b-9ba4-e94ebb3575ca" containerName="kube-rbac-proxy" Nov 03 20:40:56.559781 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.558969 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="34de3dcc-9aa3-4c4a-b827-d8e944c3514c" containerName="config-reloader" Nov 03 20:40:56.559781 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.558977 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="34de3dcc-9aa3-4c4a-b827-d8e944c3514c" containerName="thanos-sidecar" Nov 03 20:40:56.559781 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.558985 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="34de3dcc-9aa3-4c4a-b827-d8e944c3514c" containerName="kube-rbac-proxy" Nov 03 20:40:56.562816 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.562798 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:56.615630 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.615594 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 03 20:40:56.615630 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.615611 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 03 20:40:56.615630 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.615594 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 03 20:40:56.615913 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.615885 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 03 20:40:56.615913 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.615895 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 03 20:40:56.616077 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.615933 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-vj5gx\"" Nov 03 20:40:56.616133 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.616098 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 03 20:40:56.616186 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.616175 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 03 20:40:56.616308 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.616293 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 03 20:40:56.616458 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.616413 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-29m4i5anvmnkl\"" Nov 03 20:40:56.665980 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.665949 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 03 20:40:56.666151 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.666003 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 03 20:40:56.666151 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.666011 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 03 20:40:56.666151 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.666023 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 03 20:40:56.666151 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.666052 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 03 20:40:56.666151 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.666071 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 03 20:40:56.738758 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.738725 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aa9b3422-1602-42a0-a7ec-a3b520501772-web-config\") pod \"prometheus-k8s-0\" (UID: \"aa9b3422-1602-42a0-a7ec-a3b520501772\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:56.738758 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.738758 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/aa9b3422-1602-42a0-a7ec-a3b520501772-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"aa9b3422-1602-42a0-a7ec-a3b520501772\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:56.738758 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.738777 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aa9b3422-1602-42a0-a7ec-a3b520501772-config-out\") pod \"prometheus-k8s-0\" (UID: \"aa9b3422-1602-42a0-a7ec-a3b520501772\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:56.739065 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.738805 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa9b3422-1602-42a0-a7ec-a3b520501772-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aa9b3422-1602-42a0-a7ec-a3b520501772\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:56.739065 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.738877 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa9b3422-1602-42a0-a7ec-a3b520501772-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"aa9b3422-1602-42a0-a7ec-a3b520501772\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:56.739065 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.738910 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/aa9b3422-1602-42a0-a7ec-a3b520501772-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"aa9b3422-1602-42a0-a7ec-a3b520501772\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:56.739065 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.738957 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/aa9b3422-1602-42a0-a7ec-a3b520501772-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"aa9b3422-1602-42a0-a7ec-a3b520501772\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:56.739065 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.738982 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/aa9b3422-1602-42a0-a7ec-a3b520501772-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"aa9b3422-1602-42a0-a7ec-a3b520501772\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:56.739065 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.738999 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/aa9b3422-1602-42a0-a7ec-a3b520501772-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"aa9b3422-1602-42a0-a7ec-a3b520501772\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:56.739065 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.739015 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz2r6\" (UniqueName: \"kubernetes.io/projected/aa9b3422-1602-42a0-a7ec-a3b520501772-kube-api-access-bz2r6\") pod \"prometheus-k8s-0\" (UID: \"aa9b3422-1602-42a0-a7ec-a3b520501772\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:56.739065 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.739030 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/aa9b3422-1602-42a0-a7ec-a3b520501772-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"aa9b3422-1602-42a0-a7ec-a3b520501772\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:56.739360 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.739075 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/aa9b3422-1602-42a0-a7ec-a3b520501772-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"aa9b3422-1602-42a0-a7ec-a3b520501772\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:56.739360 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.739172 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/aa9b3422-1602-42a0-a7ec-a3b520501772-config\") pod \"prometheus-k8s-0\" (UID: \"aa9b3422-1602-42a0-a7ec-a3b520501772\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:56.739360 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.739193 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa9b3422-1602-42a0-a7ec-a3b520501772-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aa9b3422-1602-42a0-a7ec-a3b520501772\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:56.739360 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.739212 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aa9b3422-1602-42a0-a7ec-a3b520501772-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"aa9b3422-1602-42a0-a7ec-a3b520501772\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:56.739360 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.739226 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aa9b3422-1602-42a0-a7ec-a3b520501772-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"aa9b3422-1602-42a0-a7ec-a3b520501772\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:56.739360 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.739285 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/aa9b3422-1602-42a0-a7ec-a3b520501772-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"aa9b3422-1602-42a0-a7ec-a3b520501772\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:56.739592 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.739388 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aa9b3422-1602-42a0-a7ec-a3b520501772-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"aa9b3422-1602-42a0-a7ec-a3b520501772\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:56.739592 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.739426 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/aa9b3422-1602-42a0-a7ec-a3b520501772-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"aa9b3422-1602-42a0-a7ec-a3b520501772\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:56.739592 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.739462 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa9b3422-1602-42a0-a7ec-a3b520501772-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aa9b3422-1602-42a0-a7ec-a3b520501772\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:56.797281 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.797236 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 20:40:56.840642 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.840556 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/aa9b3422-1602-42a0-a7ec-a3b520501772-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"aa9b3422-1602-42a0-a7ec-a3b520501772\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:56.840642 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.840596 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/aa9b3422-1602-42a0-a7ec-a3b520501772-config\") pod \"prometheus-k8s-0\" (UID: \"aa9b3422-1602-42a0-a7ec-a3b520501772\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:56.840642 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.840621 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa9b3422-1602-42a0-a7ec-a3b520501772-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aa9b3422-1602-42a0-a7ec-a3b520501772\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:56.840642 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.840640 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aa9b3422-1602-42a0-a7ec-a3b520501772-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"aa9b3422-1602-42a0-a7ec-a3b520501772\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:56.840944 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.840655 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aa9b3422-1602-42a0-a7ec-a3b520501772-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"aa9b3422-1602-42a0-a7ec-a3b520501772\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:56.840944 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.840678 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/aa9b3422-1602-42a0-a7ec-a3b520501772-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"aa9b3422-1602-42a0-a7ec-a3b520501772\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:56.840944 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.840699 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aa9b3422-1602-42a0-a7ec-a3b520501772-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"aa9b3422-1602-42a0-a7ec-a3b520501772\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:56.840944 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.840722 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/aa9b3422-1602-42a0-a7ec-a3b520501772-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"aa9b3422-1602-42a0-a7ec-a3b520501772\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:56.840944 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.840742 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa9b3422-1602-42a0-a7ec-a3b520501772-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aa9b3422-1602-42a0-a7ec-a3b520501772\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:56.840944 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.840939 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aa9b3422-1602-42a0-a7ec-a3b520501772-web-config\") pod \"prometheus-k8s-0\" (UID: \"aa9b3422-1602-42a0-a7ec-a3b520501772\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:56.841215 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.840972 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/aa9b3422-1602-42a0-a7ec-a3b520501772-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"aa9b3422-1602-42a0-a7ec-a3b520501772\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:56.841215 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.841002 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aa9b3422-1602-42a0-a7ec-a3b520501772-config-out\") pod \"prometheus-k8s-0\" (UID: \"aa9b3422-1602-42a0-a7ec-a3b520501772\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:56.841215 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.841038 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa9b3422-1602-42a0-a7ec-a3b520501772-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aa9b3422-1602-42a0-a7ec-a3b520501772\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:56.841215 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.841060 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa9b3422-1602-42a0-a7ec-a3b520501772-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"aa9b3422-1602-42a0-a7ec-a3b520501772\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:56.841215 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.841084 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/aa9b3422-1602-42a0-a7ec-a3b520501772-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"aa9b3422-1602-42a0-a7ec-a3b520501772\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:56.841215 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.841105 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/aa9b3422-1602-42a0-a7ec-a3b520501772-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"aa9b3422-1602-42a0-a7ec-a3b520501772\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:56.841215 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.841147 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/aa9b3422-1602-42a0-a7ec-a3b520501772-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"aa9b3422-1602-42a0-a7ec-a3b520501772\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:56.841215 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.841175 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/aa9b3422-1602-42a0-a7ec-a3b520501772-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"aa9b3422-1602-42a0-a7ec-a3b520501772\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:56.841215 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.841199 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz2r6\" (UniqueName: \"kubernetes.io/projected/aa9b3422-1602-42a0-a7ec-a3b520501772-kube-api-access-bz2r6\") pod \"prometheus-k8s-0\" (UID: \"aa9b3422-1602-42a0-a7ec-a3b520501772\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:56.841747 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.841227 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/aa9b3422-1602-42a0-a7ec-a3b520501772-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"aa9b3422-1602-42a0-a7ec-a3b520501772\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:56.843125 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.842266 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa9b3422-1602-42a0-a7ec-a3b520501772-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aa9b3422-1602-42a0-a7ec-a3b520501772\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:56.843125 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.842774 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/aa9b3422-1602-42a0-a7ec-a3b520501772-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"aa9b3422-1602-42a0-a7ec-a3b520501772\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:56.843611 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.843427 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa9b3422-1602-42a0-a7ec-a3b520501772-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aa9b3422-1602-42a0-a7ec-a3b520501772\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:56.844294 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.843959 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/aa9b3422-1602-42a0-a7ec-a3b520501772-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"aa9b3422-1602-42a0-a7ec-a3b520501772\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:56.844380 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.844359 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aa9b3422-1602-42a0-a7ec-a3b520501772-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"aa9b3422-1602-42a0-a7ec-a3b520501772\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:56.844508 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.844478 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/aa9b3422-1602-42a0-a7ec-a3b520501772-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"aa9b3422-1602-42a0-a7ec-a3b520501772\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:56.844717 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.844690 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/aa9b3422-1602-42a0-a7ec-a3b520501772-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"aa9b3422-1602-42a0-a7ec-a3b520501772\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:56.844865 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.844847 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa9b3422-1602-42a0-a7ec-a3b520501772-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aa9b3422-1602-42a0-a7ec-a3b520501772\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:56.845085 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.845065 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/aa9b3422-1602-42a0-a7ec-a3b520501772-config\") pod \"prometheus-k8s-0\" (UID: \"aa9b3422-1602-42a0-a7ec-a3b520501772\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:56.845173 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.845123 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/aa9b3422-1602-42a0-a7ec-a3b520501772-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"aa9b3422-1602-42a0-a7ec-a3b520501772\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:56.845503 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.845477 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa9b3422-1602-42a0-a7ec-a3b520501772-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"aa9b3422-1602-42a0-a7ec-a3b520501772\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:56.845728 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.845710 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aa9b3422-1602-42a0-a7ec-a3b520501772-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"aa9b3422-1602-42a0-a7ec-a3b520501772\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:56.846312 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.846292 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/aa9b3422-1602-42a0-a7ec-a3b520501772-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"aa9b3422-1602-42a0-a7ec-a3b520501772\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:56.846784 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.846760 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aa9b3422-1602-42a0-a7ec-a3b520501772-web-config\") pod \"prometheus-k8s-0\" (UID: \"aa9b3422-1602-42a0-a7ec-a3b520501772\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:56.846869 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.846804 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aa9b3422-1602-42a0-a7ec-a3b520501772-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"aa9b3422-1602-42a0-a7ec-a3b520501772\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:56.847007 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.846988 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aa9b3422-1602-42a0-a7ec-a3b520501772-config-out\") pod \"prometheus-k8s-0\" (UID: \"aa9b3422-1602-42a0-a7ec-a3b520501772\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:56.847065 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.847047 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/aa9b3422-1602-42a0-a7ec-a3b520501772-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"aa9b3422-1602-42a0-a7ec-a3b520501772\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:56.848191 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.848169 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/aa9b3422-1602-42a0-a7ec-a3b520501772-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"aa9b3422-1602-42a0-a7ec-a3b520501772\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:56.848385 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.848367 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/aa9b3422-1602-42a0-a7ec-a3b520501772-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"aa9b3422-1602-42a0-a7ec-a3b520501772\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:56.892724 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.892692 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34de3dcc-9aa3-4c4a-b827-d8e944c3514c" path="/var/lib/kubelet/pods/34de3dcc-9aa3-4c4a-b827-d8e944c3514c/volumes" Nov 03 20:40:56.904811 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:56.904783 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz2r6\" (UniqueName: \"kubernetes.io/projected/aa9b3422-1602-42a0-a7ec-a3b520501772-kube-api-access-bz2r6\") pod \"prometheus-k8s-0\" (UID: \"aa9b3422-1602-42a0-a7ec-a3b520501772\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:57.186308 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:57.186272 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:40:57.670227 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:57.670198 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 20:40:57.673182 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:40:57.673153 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa9b3422_1602_42a0_a7ec_a3b520501772.slice/crio-9dc17e22d1064562b68906cf78711dd8fb3558a122c1e04db25ed1532b159034 WatchSource:0}: Error finding container 9dc17e22d1064562b68906cf78711dd8fb3558a122c1e04db25ed1532b159034: Status 404 returned error can't find the container with id 9dc17e22d1064562b68906cf78711dd8fb3558a122c1e04db25ed1532b159034 Nov 03 20:40:58.349641 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:58.349603 4305 generic.go:356] "Generic (PLEG): container finished" podID="aa9b3422-1602-42a0-a7ec-a3b520501772" containerID="64c692df86c85df7fa747b15ec75557d2e9ba3b227d15fcf21122fda37a8db93" exitCode=0 Nov 03 20:40:58.349803 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:58.349687 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aa9b3422-1602-42a0-a7ec-a3b520501772","Type":"ContainerDied","Data":"64c692df86c85df7fa747b15ec75557d2e9ba3b227d15fcf21122fda37a8db93"} Nov 03 20:40:58.349803 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:58.349747 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aa9b3422-1602-42a0-a7ec-a3b520501772","Type":"ContainerStarted","Data":"9dc17e22d1064562b68906cf78711dd8fb3558a122c1e04db25ed1532b159034"} Nov 03 20:40:59.356774 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:59.356740 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aa9b3422-1602-42a0-a7ec-a3b520501772","Type":"ContainerStarted","Data":"c05c974ea24c403e6da46826cfd259c0ee7f20832b3d71c4c892693468f7eaec"} Nov 03 20:40:59.357279 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:59.356816 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aa9b3422-1602-42a0-a7ec-a3b520501772","Type":"ContainerStarted","Data":"fbd412c72f65be01b5608cdef5dffdf7c9a5afba0bf10727391ae07a920ff6fc"} Nov 03 20:40:59.357279 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:59.356855 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aa9b3422-1602-42a0-a7ec-a3b520501772","Type":"ContainerStarted","Data":"c3ccb264d4ec6920eba76264a4488618f4266de508ca5cc3e4892ab60242994a"} Nov 03 20:40:59.357279 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:59.356897 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aa9b3422-1602-42a0-a7ec-a3b520501772","Type":"ContainerStarted","Data":"14f097c13e19bc39f29851fb2f98eb36068576de70e9035756df69ed0ace570f"} Nov 03 20:40:59.357279 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:59.356941 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aa9b3422-1602-42a0-a7ec-a3b520501772","Type":"ContainerStarted","Data":"9f65a08089f6adaffe1bb38225f7fe0a8b4f20d093b80ee9cd16054fbe1fb6b9"} Nov 03 20:40:59.357279 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:59.356978 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aa9b3422-1602-42a0-a7ec-a3b520501772","Type":"ContainerStarted","Data":"76e4465e08d800d70283ce0930a278090cdecf71ba19c2c1a7e501200c5c5016"} Nov 03 20:40:59.489913 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:40:59.489858 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.47305613 podStartE2EDuration="3.489837608s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.781478ms" isStatefulPod=false podCreationTimestamp="2026-11-03 20:40:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:40:59.488277976 +0000 UTC m=+94.713597780" watchObservedRunningTime="2026-11-03 20:40:59.489837608 +0000 UTC m=+94.715157398" Nov 03 20:41:02.186724 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:41:02.186681 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:41:16.320858 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:41:16.320817 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 20:41:46.319750 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:41:46.319665 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 20:41:57.187390 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:41:57.187342 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:41:57.204082 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:41:57.204046 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:41:57.581675 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:41:57.581588 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:42:16.319929 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:42:16.319891 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 20:42:46.319841 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:42:46.319792 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 20:43:16.321917 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:43:16.321877 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 20:43:28.426562 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:28.426513 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd"] Nov 03 20:43:28.429768 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:28.429750 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd" Nov 03 20:43:28.482002 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:28.481973 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 20:43:28.489488 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:28.489455 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c97844f3-937c-4873-865a-c408d83b0890-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd\" (UID: \"c97844f3-937c-4873-865a-c408d83b0890\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd" Nov 03 20:43:28.489678 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:28.489508 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c97844f3-937c-4873-865a-c408d83b0890-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd\" (UID: \"c97844f3-937c-4873-865a-c408d83b0890\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd" Nov 03 20:43:28.489678 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:28.489575 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c97844f3-937c-4873-865a-c408d83b0890-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd\" (UID: \"c97844f3-937c-4873-865a-c408d83b0890\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd" Nov 03 20:43:28.489678 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:28.489605 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c97844f3-937c-4873-865a-c408d83b0890-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd\" (UID: \"c97844f3-937c-4873-865a-c408d83b0890\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd" Nov 03 20:43:28.489678 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:28.489631 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c97844f3-937c-4873-865a-c408d83b0890-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd\" (UID: \"c97844f3-937c-4873-865a-c408d83b0890\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd" Nov 03 20:43:28.489833 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:28.489684 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdrmz\" (UniqueName: \"kubernetes.io/projected/c97844f3-937c-4873-865a-c408d83b0890-kube-api-access-sdrmz\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd\" (UID: \"c97844f3-937c-4873-865a-c408d83b0890\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd" Nov 03 20:43:28.489833 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:28.489741 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c97844f3-937c-4873-865a-c408d83b0890-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd\" (UID: \"c97844f3-937c-4873-865a-c408d83b0890\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd" Nov 03 20:43:28.489833 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:28.489788 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c97844f3-937c-4873-865a-c408d83b0890-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd\" (UID: \"c97844f3-937c-4873-865a-c408d83b0890\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd" Nov 03 20:43:28.489833 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:28.489826 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c97844f3-937c-4873-865a-c408d83b0890-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd\" (UID: \"c97844f3-937c-4873-865a-c408d83b0890\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd" Nov 03 20:43:28.498348 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:28.498326 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-mmsql\"" Nov 03 20:43:28.531536 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:28.531498 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 20:43:28.531536 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:28.531498 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 20:43:28.532957 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:28.532932 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd"] Nov 03 20:43:28.590970 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:28.590937 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c97844f3-937c-4873-865a-c408d83b0890-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd\" (UID: \"c97844f3-937c-4873-865a-c408d83b0890\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd" Nov 03 20:43:28.591179 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:28.590990 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c97844f3-937c-4873-865a-c408d83b0890-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd\" (UID: \"c97844f3-937c-4873-865a-c408d83b0890\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd" Nov 03 20:43:28.591179 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:28.591006 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c97844f3-937c-4873-865a-c408d83b0890-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd\" (UID: \"c97844f3-937c-4873-865a-c408d83b0890\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd" Nov 03 20:43:28.591179 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:28.591026 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c97844f3-937c-4873-865a-c408d83b0890-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd\" (UID: \"c97844f3-937c-4873-865a-c408d83b0890\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd" Nov 03 20:43:28.591179 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:28.591144 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c97844f3-937c-4873-865a-c408d83b0890-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd\" (UID: \"c97844f3-937c-4873-865a-c408d83b0890\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd" Nov 03 20:43:28.591502 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:28.591186 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdrmz\" (UniqueName: \"kubernetes.io/projected/c97844f3-937c-4873-865a-c408d83b0890-kube-api-access-sdrmz\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd\" (UID: \"c97844f3-937c-4873-865a-c408d83b0890\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd" Nov 03 20:43:28.591502 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:28.591207 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c97844f3-937c-4873-865a-c408d83b0890-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd\" (UID: \"c97844f3-937c-4873-865a-c408d83b0890\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd" Nov 03 20:43:28.591502 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:28.591237 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c97844f3-937c-4873-865a-c408d83b0890-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd\" (UID: \"c97844f3-937c-4873-865a-c408d83b0890\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd" Nov 03 20:43:28.591502 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:28.591452 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c97844f3-937c-4873-865a-c408d83b0890-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd\" (UID: \"c97844f3-937c-4873-865a-c408d83b0890\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd" Nov 03 20:43:28.591502 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:28.591477 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c97844f3-937c-4873-865a-c408d83b0890-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd\" (UID: \"c97844f3-937c-4873-865a-c408d83b0890\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd" Nov 03 20:43:28.591825 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:28.591801 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c97844f3-937c-4873-865a-c408d83b0890-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd\" (UID: \"c97844f3-937c-4873-865a-c408d83b0890\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd" Nov 03 20:43:28.591889 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:28.591820 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c97844f3-937c-4873-865a-c408d83b0890-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd\" (UID: \"c97844f3-937c-4873-865a-c408d83b0890\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd" Nov 03 20:43:28.591889 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:28.591843 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c97844f3-937c-4873-865a-c408d83b0890-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd\" (UID: \"c97844f3-937c-4873-865a-c408d83b0890\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd" Nov 03 20:43:28.591990 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:28.591973 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c97844f3-937c-4873-865a-c408d83b0890-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd\" (UID: \"c97844f3-937c-4873-865a-c408d83b0890\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd" Nov 03 20:43:28.593512 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:28.593492 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c97844f3-937c-4873-865a-c408d83b0890-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd\" (UID: \"c97844f3-937c-4873-865a-c408d83b0890\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd" Nov 03 20:43:28.593752 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:28.593736 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c97844f3-937c-4873-865a-c408d83b0890-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd\" (UID: \"c97844f3-937c-4873-865a-c408d83b0890\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd" Nov 03 20:43:28.673079 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:28.673035 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdrmz\" (UniqueName: \"kubernetes.io/projected/c97844f3-937c-4873-865a-c408d83b0890-kube-api-access-sdrmz\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd\" (UID: \"c97844f3-937c-4873-865a-c408d83b0890\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd" Nov 03 20:43:28.673693 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:28.673619 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c97844f3-937c-4873-865a-c408d83b0890-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd\" (UID: \"c97844f3-937c-4873-865a-c408d83b0890\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd" Nov 03 20:43:28.730965 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:28.730889 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr"] Nov 03 20:43:28.735126 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:28.735106 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr" Nov 03 20:43:28.741779 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:28.741755 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd" Nov 03 20:43:28.787253 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:28.787218 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-6lktv\"" Nov 03 20:43:28.797996 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:28.797818 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr"] Nov 03 20:43:28.798815 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:28.798640 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/148ae545-6008-4fd7-b538-eebc3a5e447f-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr\" (UID: \"148ae545-6008-4fd7-b538-eebc3a5e447f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr" Nov 03 20:43:28.798815 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:28.798681 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/148ae545-6008-4fd7-b538-eebc3a5e447f-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr\" (UID: \"148ae545-6008-4fd7-b538-eebc3a5e447f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr" Nov 03 20:43:28.798815 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:28.798700 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/148ae545-6008-4fd7-b538-eebc3a5e447f-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr\" (UID: \"148ae545-6008-4fd7-b538-eebc3a5e447f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr" Nov 03 20:43:28.798815 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:28.798720 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/148ae545-6008-4fd7-b538-eebc3a5e447f-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr\" (UID: \"148ae545-6008-4fd7-b538-eebc3a5e447f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr" Nov 03 20:43:28.798815 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:28.798761 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzshh\" (UniqueName: \"kubernetes.io/projected/148ae545-6008-4fd7-b538-eebc3a5e447f-kube-api-access-wzshh\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr\" (UID: \"148ae545-6008-4fd7-b538-eebc3a5e447f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr" Nov 03 20:43:28.799092 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:28.798861 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/148ae545-6008-4fd7-b538-eebc3a5e447f-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr\" (UID: \"148ae545-6008-4fd7-b538-eebc3a5e447f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr" Nov 03 20:43:28.799092 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:28.798946 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/148ae545-6008-4fd7-b538-eebc3a5e447f-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr\" (UID: \"148ae545-6008-4fd7-b538-eebc3a5e447f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr" Nov 03 20:43:28.799092 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:28.798968 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/148ae545-6008-4fd7-b538-eebc3a5e447f-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr\" (UID: \"148ae545-6008-4fd7-b538-eebc3a5e447f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr" Nov 03 20:43:28.799092 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:28.798987 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/148ae545-6008-4fd7-b538-eebc3a5e447f-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr\" (UID: \"148ae545-6008-4fd7-b538-eebc3a5e447f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr" Nov 03 20:43:28.900312 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:28.900277 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/148ae545-6008-4fd7-b538-eebc3a5e447f-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr\" (UID: \"148ae545-6008-4fd7-b538-eebc3a5e447f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr" Nov 03 20:43:28.900538 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:28.900324 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/148ae545-6008-4fd7-b538-eebc3a5e447f-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr\" (UID: \"148ae545-6008-4fd7-b538-eebc3a5e447f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr" Nov 03 20:43:28.900538 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:28.900474 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/148ae545-6008-4fd7-b538-eebc3a5e447f-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr\" (UID: \"148ae545-6008-4fd7-b538-eebc3a5e447f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr" Nov 03 20:43:28.900538 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:28.900519 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/148ae545-6008-4fd7-b538-eebc3a5e447f-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr\" (UID: \"148ae545-6008-4fd7-b538-eebc3a5e447f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr" Nov 03 20:43:28.900720 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:28.900551 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/148ae545-6008-4fd7-b538-eebc3a5e447f-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr\" (UID: \"148ae545-6008-4fd7-b538-eebc3a5e447f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr" Nov 03 20:43:28.900720 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:28.900605 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/148ae545-6008-4fd7-b538-eebc3a5e447f-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr\" (UID: \"148ae545-6008-4fd7-b538-eebc3a5e447f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr" Nov 03 20:43:28.900720 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:28.900634 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/148ae545-6008-4fd7-b538-eebc3a5e447f-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr\" (UID: \"148ae545-6008-4fd7-b538-eebc3a5e447f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr" Nov 03 20:43:28.900720 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:28.900671 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/148ae545-6008-4fd7-b538-eebc3a5e447f-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr\" (UID: \"148ae545-6008-4fd7-b538-eebc3a5e447f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr" Nov 03 20:43:28.900900 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:28.900716 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/148ae545-6008-4fd7-b538-eebc3a5e447f-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr\" (UID: \"148ae545-6008-4fd7-b538-eebc3a5e447f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr" Nov 03 20:43:28.900900 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:28.900730 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzshh\" (UniqueName: \"kubernetes.io/projected/148ae545-6008-4fd7-b538-eebc3a5e447f-kube-api-access-wzshh\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr\" (UID: \"148ae545-6008-4fd7-b538-eebc3a5e447f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr" Nov 03 20:43:28.901168 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:28.901098 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/148ae545-6008-4fd7-b538-eebc3a5e447f-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr\" (UID: \"148ae545-6008-4fd7-b538-eebc3a5e447f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr" Nov 03 20:43:28.901295 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:28.901221 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/148ae545-6008-4fd7-b538-eebc3a5e447f-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr\" (UID: \"148ae545-6008-4fd7-b538-eebc3a5e447f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr" Nov 03 20:43:28.901295 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:28.901285 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/148ae545-6008-4fd7-b538-eebc3a5e447f-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr\" (UID: \"148ae545-6008-4fd7-b538-eebc3a5e447f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr" Nov 03 20:43:28.901387 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:28.901352 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/148ae545-6008-4fd7-b538-eebc3a5e447f-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr\" (UID: \"148ae545-6008-4fd7-b538-eebc3a5e447f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr" Nov 03 20:43:28.902825 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:28.902805 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/148ae545-6008-4fd7-b538-eebc3a5e447f-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr\" (UID: \"148ae545-6008-4fd7-b538-eebc3a5e447f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr" Nov 03 20:43:28.902952 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:28.902935 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/148ae545-6008-4fd7-b538-eebc3a5e447f-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr\" (UID: \"148ae545-6008-4fd7-b538-eebc3a5e447f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr" Nov 03 20:43:28.962394 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:28.962358 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/148ae545-6008-4fd7-b538-eebc3a5e447f-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr\" (UID: \"148ae545-6008-4fd7-b538-eebc3a5e447f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr" Nov 03 20:43:28.962568 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:28.962498 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzshh\" (UniqueName: \"kubernetes.io/projected/148ae545-6008-4fd7-b538-eebc3a5e447f-kube-api-access-wzshh\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr\" (UID: \"148ae545-6008-4fd7-b538-eebc3a5e447f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr" Nov 03 20:43:29.048667 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:29.048575 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr" Nov 03 20:43:29.168903 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:29.168706 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd"] Nov 03 20:43:29.171426 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:43:29.171401 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc97844f3_937c_4873_865a_c408d83b0890.slice/crio-f6f792c21e6a72d1a7fc89f90ea85956f50744101de86df2e1bcc1f4c8b28eeb WatchSource:0}: Error finding container f6f792c21e6a72d1a7fc89f90ea85956f50744101de86df2e1bcc1f4c8b28eeb: Status 404 returned error can't find the container with id f6f792c21e6a72d1a7fc89f90ea85956f50744101de86df2e1bcc1f4c8b28eeb Nov 03 20:43:29.427292 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:29.427268 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr"] Nov 03 20:43:29.429727 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:43:29.429677 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod148ae545_6008_4fd7_b538_eebc3a5e447f.slice/crio-27e35d8d8e3235e867047a0ed417481f813f5661580fd2999e2d0ca7c041c260 WatchSource:0}: Error finding container 27e35d8d8e3235e867047a0ed417481f813f5661580fd2999e2d0ca7c041c260: Status 404 returned error can't find the container with id 27e35d8d8e3235e867047a0ed417481f813f5661580fd2999e2d0ca7c041c260 Nov 03 20:43:29.889962 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:29.889870 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr" event={"ID":"148ae545-6008-4fd7-b538-eebc3a5e447f","Type":"ContainerStarted","Data":"27e35d8d8e3235e867047a0ed417481f813f5661580fd2999e2d0ca7c041c260"} Nov 03 20:43:29.890994 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:29.890974 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd" event={"ID":"c97844f3-937c-4873-865a-c408d83b0890","Type":"ContainerStarted","Data":"f6f792c21e6a72d1a7fc89f90ea85956f50744101de86df2e1bcc1f4c8b28eeb"} Nov 03 20:43:33.989610 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:33.989568 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:43:33.989927 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:33.989663 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:43:33.989927 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:33.989705 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:43:34.770126 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:34.770088 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:43:34.770252 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:34.770161 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:43:34.770252 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:34.770189 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:43:34.912584 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:34.912544 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr" event={"ID":"148ae545-6008-4fd7-b538-eebc3a5e447f","Type":"ContainerStarted","Data":"435d52c2412b152b99cf642422ed4ce6826fe671061d20d3ed6d550f9986cb2d"} Nov 03 20:43:34.914278 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:34.914248 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd" event={"ID":"c97844f3-937c-4873-865a-c408d83b0890","Type":"ContainerStarted","Data":"71272b9187b8c275851e3b0e36a1ba0cbbf1f181207413e4bf02904c4edd0d07"} Nov 03 20:43:35.049159 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:35.049079 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr" Nov 03 20:43:35.053618 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:35.053570 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr" podStartSLOduration=2.496725857 podStartE2EDuration="7.053554585s" totalImagesPullingTime="4.556828728s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:43:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:43:35.051500528 +0000 UTC m=+250.276820320" watchObservedRunningTime="2026-11-03 20:43:35.053554585 +0000 UTC m=+250.278874376" Nov 03 20:43:35.054368 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:35.054348 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr" Nov 03 20:43:35.172050 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:35.171987 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd" podStartSLOduration=1.575987204 podStartE2EDuration="7.171967929s" totalImagesPullingTime="5.595980725s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:43:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:43:35.170723017 +0000 UTC m=+250.396042808" watchObservedRunningTime="2026-11-03 20:43:35.171967929 +0000 UTC m=+250.397287721" Nov 03 20:43:35.742385 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:35.742342 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd" Nov 03 20:43:35.747066 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:35.747043 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd" Nov 03 20:43:35.918456 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:35.918413 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd" Nov 03 20:43:35.918645 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:35.918501 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr" Nov 03 20:43:35.919162 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:35.919141 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd" Nov 03 20:43:35.919278 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:43:35.919250 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr" Nov 03 20:43:46.321214 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:43:46.321174 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 20:44:14.481533 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:14.481488 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-78789f7d7-6hqw2"] Nov 03 20:44:14.484884 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:14.484860 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78789f7d7-6hqw2" Nov 03 20:44:14.499677 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:14.499652 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5a029be1-8a5c-4ef2-8b9b-0e78c1fb5a22-metrics-cert\") pod \"opendatahub-operator-controller-manager-78789f7d7-6hqw2\" (UID: \"5a029be1-8a5c-4ef2-8b9b-0e78c1fb5a22\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78789f7d7-6hqw2" Nov 03 20:44:14.499847 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:14.499700 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5a029be1-8a5c-4ef2-8b9b-0e78c1fb5a22-apiservice-cert\") pod \"opendatahub-operator-controller-manager-78789f7d7-6hqw2\" (UID: \"5a029be1-8a5c-4ef2-8b9b-0e78c1fb5a22\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78789f7d7-6hqw2" Nov 03 20:44:14.499847 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:14.499723 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a029be1-8a5c-4ef2-8b9b-0e78c1fb5a22-tmp\") pod \"opendatahub-operator-controller-manager-78789f7d7-6hqw2\" (UID: \"5a029be1-8a5c-4ef2-8b9b-0e78c1fb5a22\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78789f7d7-6hqw2" Nov 03 20:44:14.499847 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:14.499743 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5a029be1-8a5c-4ef2-8b9b-0e78c1fb5a22-webhook-cert\") pod \"opendatahub-operator-controller-manager-78789f7d7-6hqw2\" (UID: \"5a029be1-8a5c-4ef2-8b9b-0e78c1fb5a22\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78789f7d7-6hqw2" Nov 03 20:44:14.499847 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:14.499762 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5a029be1-8a5c-4ef2-8b9b-0e78c1fb5a22-manifests\") pod \"opendatahub-operator-controller-manager-78789f7d7-6hqw2\" (UID: \"5a029be1-8a5c-4ef2-8b9b-0e78c1fb5a22\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78789f7d7-6hqw2" Nov 03 20:44:14.499994 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:14.499911 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7nqh\" (UniqueName: \"kubernetes.io/projected/5a029be1-8a5c-4ef2-8b9b-0e78c1fb5a22-kube-api-access-v7nqh\") pod \"opendatahub-operator-controller-manager-78789f7d7-6hqw2\" (UID: \"5a029be1-8a5c-4ef2-8b9b-0e78c1fb5a22\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78789f7d7-6hqw2" Nov 03 20:44:14.597125 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:14.597089 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-78789f7d7-6hqw2"] Nov 03 20:44:14.599648 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:14.599617 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 20:44:14.599804 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:14.599692 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 20:44:14.599804 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:14.599714 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 20:44:14.599804 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:14.599754 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 20:44:14.599929 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:14.599826 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-7q7qx\"" Nov 03 20:44:14.599929 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:14.599889 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 20:44:14.600320 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:14.600298 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5a029be1-8a5c-4ef2-8b9b-0e78c1fb5a22-apiservice-cert\") pod \"opendatahub-operator-controller-manager-78789f7d7-6hqw2\" (UID: \"5a029be1-8a5c-4ef2-8b9b-0e78c1fb5a22\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78789f7d7-6hqw2" Nov 03 20:44:14.600401 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:14.600336 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a029be1-8a5c-4ef2-8b9b-0e78c1fb5a22-tmp\") pod \"opendatahub-operator-controller-manager-78789f7d7-6hqw2\" (UID: \"5a029be1-8a5c-4ef2-8b9b-0e78c1fb5a22\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78789f7d7-6hqw2" Nov 03 20:44:14.600527 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:14.600491 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5a029be1-8a5c-4ef2-8b9b-0e78c1fb5a22-webhook-cert\") pod \"opendatahub-operator-controller-manager-78789f7d7-6hqw2\" (UID: \"5a029be1-8a5c-4ef2-8b9b-0e78c1fb5a22\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78789f7d7-6hqw2" Nov 03 20:44:14.600649 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:14.600537 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5a029be1-8a5c-4ef2-8b9b-0e78c1fb5a22-manifests\") pod \"opendatahub-operator-controller-manager-78789f7d7-6hqw2\" (UID: \"5a029be1-8a5c-4ef2-8b9b-0e78c1fb5a22\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78789f7d7-6hqw2" Nov 03 20:44:14.600649 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:14.600586 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7nqh\" (UniqueName: \"kubernetes.io/projected/5a029be1-8a5c-4ef2-8b9b-0e78c1fb5a22-kube-api-access-v7nqh\") pod \"opendatahub-operator-controller-manager-78789f7d7-6hqw2\" (UID: \"5a029be1-8a5c-4ef2-8b9b-0e78c1fb5a22\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78789f7d7-6hqw2" Nov 03 20:44:14.600649 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:14.600641 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5a029be1-8a5c-4ef2-8b9b-0e78c1fb5a22-metrics-cert\") pod \"opendatahub-operator-controller-manager-78789f7d7-6hqw2\" (UID: \"5a029be1-8a5c-4ef2-8b9b-0e78c1fb5a22\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78789f7d7-6hqw2" Nov 03 20:44:14.601294 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:14.601273 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5a029be1-8a5c-4ef2-8b9b-0e78c1fb5a22-manifests\") pod \"opendatahub-operator-controller-manager-78789f7d7-6hqw2\" (UID: \"5a029be1-8a5c-4ef2-8b9b-0e78c1fb5a22\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78789f7d7-6hqw2" Nov 03 20:44:14.603190 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:14.603167 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a029be1-8a5c-4ef2-8b9b-0e78c1fb5a22-tmp\") pod \"opendatahub-operator-controller-manager-78789f7d7-6hqw2\" (UID: \"5a029be1-8a5c-4ef2-8b9b-0e78c1fb5a22\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78789f7d7-6hqw2" Nov 03 20:44:14.603617 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:14.603590 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5a029be1-8a5c-4ef2-8b9b-0e78c1fb5a22-metrics-cert\") pod \"opendatahub-operator-controller-manager-78789f7d7-6hqw2\" (UID: \"5a029be1-8a5c-4ef2-8b9b-0e78c1fb5a22\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78789f7d7-6hqw2" Nov 03 20:44:14.603707 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:14.603635 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5a029be1-8a5c-4ef2-8b9b-0e78c1fb5a22-apiservice-cert\") pod \"opendatahub-operator-controller-manager-78789f7d7-6hqw2\" (UID: \"5a029be1-8a5c-4ef2-8b9b-0e78c1fb5a22\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78789f7d7-6hqw2" Nov 03 20:44:14.603791 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:14.603770 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5a029be1-8a5c-4ef2-8b9b-0e78c1fb5a22-webhook-cert\") pod \"opendatahub-operator-controller-manager-78789f7d7-6hqw2\" (UID: \"5a029be1-8a5c-4ef2-8b9b-0e78c1fb5a22\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78789f7d7-6hqw2" Nov 03 20:44:14.662838 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:14.662798 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7nqh\" (UniqueName: \"kubernetes.io/projected/5a029be1-8a5c-4ef2-8b9b-0e78c1fb5a22-kube-api-access-v7nqh\") pod \"opendatahub-operator-controller-manager-78789f7d7-6hqw2\" (UID: \"5a029be1-8a5c-4ef2-8b9b-0e78c1fb5a22\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78789f7d7-6hqw2" Nov 03 20:44:14.798925 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:14.798826 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78789f7d7-6hqw2" Nov 03 20:44:15.314714 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:15.314688 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-78789f7d7-6hqw2"] Nov 03 20:44:15.316987 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:44:15.316961 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a029be1_8a5c_4ef2_8b9b_0e78c1fb5a22.slice/crio-228b47af31bfa0d4fbf3aba7209aebb2cb00699623bd4b0b988fe78ced259b80 WatchSource:0}: Error finding container 228b47af31bfa0d4fbf3aba7209aebb2cb00699623bd4b0b988fe78ced259b80: Status 404 returned error can't find the container with id 228b47af31bfa0d4fbf3aba7209aebb2cb00699623bd4b0b988fe78ced259b80 Nov 03 20:44:16.068268 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:16.068233 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78789f7d7-6hqw2" event={"ID":"5a029be1-8a5c-4ef2-8b9b-0e78c1fb5a22","Type":"ContainerStarted","Data":"228b47af31bfa0d4fbf3aba7209aebb2cb00699623bd4b0b988fe78ced259b80"} Nov 03 20:44:16.322735 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:44:16.322655 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 20:44:19.084087 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:19.084055 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78789f7d7-6hqw2" event={"ID":"5a029be1-8a5c-4ef2-8b9b-0e78c1fb5a22","Type":"ContainerStarted","Data":"67c04f533e7653a94b551054bf1ab9e2973f060c428ad58a34c6e84117f9f003"} Nov 03 20:44:20.105096 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:20.105054 4305 generic.go:356] "Generic (PLEG): container finished" podID="5a029be1-8a5c-4ef2-8b9b-0e78c1fb5a22" containerID="67c04f533e7653a94b551054bf1ab9e2973f060c428ad58a34c6e84117f9f003" exitCode=0 Nov 03 20:44:20.105550 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:20.105147 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78789f7d7-6hqw2" event={"ID":"5a029be1-8a5c-4ef2-8b9b-0e78c1fb5a22","Type":"ContainerDied","Data":"67c04f533e7653a94b551054bf1ab9e2973f060c428ad58a34c6e84117f9f003"} Nov 03 20:44:21.110335 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:21.110299 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78789f7d7-6hqw2" event={"ID":"5a029be1-8a5c-4ef2-8b9b-0e78c1fb5a22","Type":"ContainerStarted","Data":"14970ff37610a9fbe42e3c25e133c26393e74b89eb1b2049505a2a9c12a8e0e9"} Nov 03 20:44:21.110760 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:21.110464 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78789f7d7-6hqw2" Nov 03 20:44:29.122249 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:29.122185 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78789f7d7-6hqw2" podStartSLOduration=10.343456285 podStartE2EDuration="15.122166776s" totalImagesPullingTime="4.348695517s" totalInitContainerRuntime="430.014974ms" isStatefulPod=false podCreationTimestamp="2026-11-03 20:44:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:44:21.37494384 +0000 UTC m=+296.600263641" watchObservedRunningTime="2026-11-03 20:44:29.122166776 +0000 UTC m=+304.347486567" Nov 03 20:44:29.122794 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:29.122778 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-wdh5m"] Nov 03 20:44:29.129248 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:29.129220 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wdh5m" Nov 03 20:44:29.198300 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:29.198262 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-wdh5m"] Nov 03 20:44:29.211081 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:29.211049 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 03 20:44:29.229770 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:29.229728 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/69ca48b7-91ec-4c65-bd60-eeca4208940b-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-wdh5m\" (UID: \"69ca48b7-91ec-4c65-bd60-eeca4208940b\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wdh5m" Nov 03 20:44:29.229953 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:29.229801 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr656\" (UniqueName: \"kubernetes.io/projected/69ca48b7-91ec-4c65-bd60-eeca4208940b-kube-api-access-nr656\") pod \"cert-manager-cainjector-5c57bbd4fc-wdh5m\" (UID: \"69ca48b7-91ec-4c65-bd60-eeca4208940b\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wdh5m" Nov 03 20:44:29.246320 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:29.246286 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-kmpn9\"" Nov 03 20:44:29.260507 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:29.260486 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 03 20:44:29.330813 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:29.330778 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr656\" (UniqueName: \"kubernetes.io/projected/69ca48b7-91ec-4c65-bd60-eeca4208940b-kube-api-access-nr656\") pod \"cert-manager-cainjector-5c57bbd4fc-wdh5m\" (UID: \"69ca48b7-91ec-4c65-bd60-eeca4208940b\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wdh5m" Nov 03 20:44:29.330990 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:29.330876 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/69ca48b7-91ec-4c65-bd60-eeca4208940b-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-wdh5m\" (UID: \"69ca48b7-91ec-4c65-bd60-eeca4208940b\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wdh5m" Nov 03 20:44:29.392464 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:29.392420 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/69ca48b7-91ec-4c65-bd60-eeca4208940b-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-wdh5m\" (UID: \"69ca48b7-91ec-4c65-bd60-eeca4208940b\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wdh5m" Nov 03 20:44:29.392726 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:29.392703 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr656\" (UniqueName: \"kubernetes.io/projected/69ca48b7-91ec-4c65-bd60-eeca4208940b-kube-api-access-nr656\") pod \"cert-manager-cainjector-5c57bbd4fc-wdh5m\" (UID: \"69ca48b7-91ec-4c65-bd60-eeca4208940b\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wdh5m" Nov 03 20:44:29.450741 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:29.450702 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wdh5m" Nov 03 20:44:29.849515 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:29.849290 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-wdh5m"] Nov 03 20:44:29.852103 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:44:29.852079 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod69ca48b7_91ec_4c65_bd60_eeca4208940b.slice/crio-e6a9efc9b07dd599eb52c5175f1f92a02019aaf166ceb1096671ad6a7972231c WatchSource:0}: Error finding container e6a9efc9b07dd599eb52c5175f1f92a02019aaf166ceb1096671ad6a7972231c: Status 404 returned error can't find the container with id e6a9efc9b07dd599eb52c5175f1f92a02019aaf166ceb1096671ad6a7972231c Nov 03 20:44:29.854480 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:29.854465 4305 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:44:30.146394 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:30.146361 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wdh5m" event={"ID":"69ca48b7-91ec-4c65-bd60-eeca4208940b","Type":"ContainerStarted","Data":"e6a9efc9b07dd599eb52c5175f1f92a02019aaf166ceb1096671ad6a7972231c"} Nov 03 20:44:32.117141 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:32.117110 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78789f7d7-6hqw2" Nov 03 20:44:34.165743 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:34.165710 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wdh5m" event={"ID":"69ca48b7-91ec-4c65-bd60-eeca4208940b","Type":"ContainerStarted","Data":"6613181ad52eaf9a5b1e3bd89b1a97473bd3b406a3c3561535f4ab8655f07e3d"} Nov 03 20:44:38.372257 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:38.372150 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wdh5m" podStartSLOduration=5.433857376 podStartE2EDuration="9.372135592s" totalImagesPullingTime="3.938278216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:44:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:44:34.286829562 +0000 UTC m=+309.512149352" watchObservedRunningTime="2026-11-03 20:44:38.372135592 +0000 UTC m=+313.597455384" Nov 03 20:44:38.372653 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:38.372366 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vhvmb"] Nov 03 20:44:38.375569 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:38.375553 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vhvmb" Nov 03 20:44:38.432395 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:38.432359 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vhvmb"] Nov 03 20:44:38.480196 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:38.480158 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 20:44:38.480196 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:38.480167 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 20:44:38.480419 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:38.480232 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-6p7qs\"" Nov 03 20:44:38.521610 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:38.521573 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/089f2b4a-9ad0-4e71-9efe-a22df561cc8f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-vhvmb\" (UID: \"089f2b4a-9ad0-4e71-9efe-a22df561cc8f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vhvmb" Nov 03 20:44:38.521779 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:38.521682 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh8lz\" (UniqueName: \"kubernetes.io/projected/089f2b4a-9ad0-4e71-9efe-a22df561cc8f-kube-api-access-mh8lz\") pod \"kuadrant-operator-controller-manager-cf8469d7b-vhvmb\" (UID: \"089f2b4a-9ad0-4e71-9efe-a22df561cc8f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vhvmb" Nov 03 20:44:38.622347 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:38.622253 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/089f2b4a-9ad0-4e71-9efe-a22df561cc8f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-vhvmb\" (UID: \"089f2b4a-9ad0-4e71-9efe-a22df561cc8f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vhvmb" Nov 03 20:44:38.622347 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:38.622336 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh8lz\" (UniqueName: \"kubernetes.io/projected/089f2b4a-9ad0-4e71-9efe-a22df561cc8f-kube-api-access-mh8lz\") pod \"kuadrant-operator-controller-manager-cf8469d7b-vhvmb\" (UID: \"089f2b4a-9ad0-4e71-9efe-a22df561cc8f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vhvmb" Nov 03 20:44:38.622704 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:38.622684 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/089f2b4a-9ad0-4e71-9efe-a22df561cc8f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-vhvmb\" (UID: \"089f2b4a-9ad0-4e71-9efe-a22df561cc8f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vhvmb" Nov 03 20:44:38.688679 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:38.688645 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh8lz\" (UniqueName: \"kubernetes.io/projected/089f2b4a-9ad0-4e71-9efe-a22df561cc8f-kube-api-access-mh8lz\") pod \"kuadrant-operator-controller-manager-cf8469d7b-vhvmb\" (UID: \"089f2b4a-9ad0-4e71-9efe-a22df561cc8f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vhvmb" Nov 03 20:44:38.987837 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:38.987784 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vhvmb" Nov 03 20:44:39.483971 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:44:39.483938 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod089f2b4a_9ad0_4e71_9efe_a22df561cc8f.slice/crio-56daade9df409a5b09f04df94f0070cf9a46c635c783a2ff317878567adb4e55 WatchSource:0}: Error finding container 56daade9df409a5b09f04df94f0070cf9a46c635c783a2ff317878567adb4e55: Status 404 returned error can't find the container with id 56daade9df409a5b09f04df94f0070cf9a46c635c783a2ff317878567adb4e55 Nov 03 20:44:39.527194 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:39.527157 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vhvmb"] Nov 03 20:44:40.191785 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:40.191747 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vhvmb" event={"ID":"089f2b4a-9ad0-4e71-9efe-a22df561cc8f","Type":"ContainerStarted","Data":"56daade9df409a5b09f04df94f0070cf9a46c635c783a2ff317878567adb4e55"} Nov 03 20:44:45.217057 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:45.217022 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vhvmb" event={"ID":"089f2b4a-9ad0-4e71-9efe-a22df561cc8f","Type":"ContainerStarted","Data":"2e6fb409335e3ad40caf77dd10cd717fd08e29ab5ffc6ab024303c4018cecddc"} Nov 03 20:44:45.217448 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:45.217127 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vhvmb" Nov 03 20:44:45.441921 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:45.441865 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vhvmb" podStartSLOduration=2.470259948 podStartE2EDuration="7.441849906s" totalImagesPullingTime="4.971589958s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:44:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:44:45.391775602 +0000 UTC m=+320.617095395" watchObservedRunningTime="2026-11-03 20:44:45.441849906 +0000 UTC m=+320.667169697" Nov 03 20:44:46.320608 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:44:46.320574 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 20:44:50.139457 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:50.136419 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-4xr8g"] Nov 03 20:44:50.141829 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:50.140294 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4xr8g" Nov 03 20:44:50.194691 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:50.194658 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Nov 03 20:44:50.228971 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:50.228926 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/393b09a4-4340-4396-82fe-f91addfa5568-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-4xr8g\" (UID: \"393b09a4-4340-4396-82fe-f91addfa5568\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4xr8g" Nov 03 20:44:50.229202 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:50.229045 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/393b09a4-4340-4396-82fe-f91addfa5568-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-4xr8g\" (UID: \"393b09a4-4340-4396-82fe-f91addfa5568\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4xr8g" Nov 03 20:44:50.229202 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:50.229145 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v5wn\" (UniqueName: \"kubernetes.io/projected/393b09a4-4340-4396-82fe-f91addfa5568-kube-api-access-8v5wn\") pod \"kuadrant-console-plugin-66c4fb575d-4xr8g\" (UID: \"393b09a4-4340-4396-82fe-f91addfa5568\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4xr8g" Nov 03 20:44:50.244146 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:50.244114 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Nov 03 20:44:50.244294 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:50.244121 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-77nnf\"" Nov 03 20:44:50.245410 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:50.245392 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-4xr8g"] Nov 03 20:44:50.329795 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:50.329762 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/393b09a4-4340-4396-82fe-f91addfa5568-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-4xr8g\" (UID: \"393b09a4-4340-4396-82fe-f91addfa5568\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4xr8g" Nov 03 20:44:50.329795 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:50.329800 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/393b09a4-4340-4396-82fe-f91addfa5568-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-4xr8g\" (UID: \"393b09a4-4340-4396-82fe-f91addfa5568\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4xr8g" Nov 03 20:44:50.330063 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:50.329866 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v5wn\" (UniqueName: \"kubernetes.io/projected/393b09a4-4340-4396-82fe-f91addfa5568-kube-api-access-8v5wn\") pod \"kuadrant-console-plugin-66c4fb575d-4xr8g\" (UID: \"393b09a4-4340-4396-82fe-f91addfa5568\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4xr8g" Nov 03 20:44:50.330063 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:44:50.329940 4305 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Nov 03 20:44:50.330063 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:44:50.330030 4305 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/393b09a4-4340-4396-82fe-f91addfa5568-plugin-serving-cert podName:393b09a4-4340-4396-82fe-f91addfa5568 nodeName:}" failed. No retries permitted until 2026-11-03 20:44:50.830006524 +0000 UTC m=+326.055326307 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/393b09a4-4340-4396-82fe-f91addfa5568-plugin-serving-cert") pod "kuadrant-console-plugin-66c4fb575d-4xr8g" (UID: "393b09a4-4340-4396-82fe-f91addfa5568") : secret "plugin-serving-cert" not found Nov 03 20:44:50.330570 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:50.330550 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/393b09a4-4340-4396-82fe-f91addfa5568-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-4xr8g\" (UID: \"393b09a4-4340-4396-82fe-f91addfa5568\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4xr8g" Nov 03 20:44:50.390930 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:50.390855 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v5wn\" (UniqueName: \"kubernetes.io/projected/393b09a4-4340-4396-82fe-f91addfa5568-kube-api-access-8v5wn\") pod \"kuadrant-console-plugin-66c4fb575d-4xr8g\" (UID: \"393b09a4-4340-4396-82fe-f91addfa5568\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4xr8g" Nov 03 20:44:50.834129 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:50.834086 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/393b09a4-4340-4396-82fe-f91addfa5568-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-4xr8g\" (UID: \"393b09a4-4340-4396-82fe-f91addfa5568\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4xr8g" Nov 03 20:44:50.836835 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:50.836807 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/393b09a4-4340-4396-82fe-f91addfa5568-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-4xr8g\" (UID: \"393b09a4-4340-4396-82fe-f91addfa5568\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4xr8g" Nov 03 20:44:51.056810 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:51.056779 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4xr8g" Nov 03 20:44:51.427311 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:51.427289 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-4xr8g"] Nov 03 20:44:51.430051 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:44:51.430021 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod393b09a4_4340_4396_82fe_f91addfa5568.slice/crio-dc71c58efa69e59e49d223aaf27d3ee6320423af19ea0e97eaefe9608a8aa4f8 WatchSource:0}: Error finding container dc71c58efa69e59e49d223aaf27d3ee6320423af19ea0e97eaefe9608a8aa4f8: Status 404 returned error can't find the container with id dc71c58efa69e59e49d223aaf27d3ee6320423af19ea0e97eaefe9608a8aa4f8 Nov 03 20:44:52.249159 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:52.249125 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4xr8g" event={"ID":"393b09a4-4340-4396-82fe-f91addfa5568","Type":"ContainerStarted","Data":"dc71c58efa69e59e49d223aaf27d3ee6320423af19ea0e97eaefe9608a8aa4f8"} Nov 03 20:44:54.860136 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:54.860094 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-68d884f4c6-jxlpw"] Nov 03 20:44:54.875257 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:54.875230 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-68d884f4c6-jxlpw" Nov 03 20:44:54.950115 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:54.950081 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-68d884f4c6-jxlpw"] Nov 03 20:44:54.975708 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:54.975671 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjpgl\" (UniqueName: \"kubernetes.io/projected/6cb31134-47e9-49b0-9412-e9dbcc5d588a-kube-api-access-xjpgl\") pod \"kube-auth-proxy-68d884f4c6-jxlpw\" (UID: \"6cb31134-47e9-49b0-9412-e9dbcc5d588a\") " pod="openshift-ingress/kube-auth-proxy-68d884f4c6-jxlpw" Nov 03 20:44:54.975914 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:54.975721 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6cb31134-47e9-49b0-9412-e9dbcc5d588a-tmp\") pod \"kube-auth-proxy-68d884f4c6-jxlpw\" (UID: \"6cb31134-47e9-49b0-9412-e9dbcc5d588a\") " pod="openshift-ingress/kube-auth-proxy-68d884f4c6-jxlpw" Nov 03 20:44:54.975914 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:54.975858 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cb31134-47e9-49b0-9412-e9dbcc5d588a-tls-certs\") pod \"kube-auth-proxy-68d884f4c6-jxlpw\" (UID: \"6cb31134-47e9-49b0-9412-e9dbcc5d588a\") " pod="openshift-ingress/kube-auth-proxy-68d884f4c6-jxlpw" Nov 03 20:44:55.000815 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:55.000776 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 03 20:44:55.040485 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:55.040453 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 03 20:44:55.040485 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:55.040456 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-wr9t8\"" Nov 03 20:44:55.077223 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:55.077184 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cb31134-47e9-49b0-9412-e9dbcc5d588a-tls-certs\") pod \"kube-auth-proxy-68d884f4c6-jxlpw\" (UID: \"6cb31134-47e9-49b0-9412-e9dbcc5d588a\") " pod="openshift-ingress/kube-auth-proxy-68d884f4c6-jxlpw" Nov 03 20:44:55.077404 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:55.077283 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjpgl\" (UniqueName: \"kubernetes.io/projected/6cb31134-47e9-49b0-9412-e9dbcc5d588a-kube-api-access-xjpgl\") pod \"kube-auth-proxy-68d884f4c6-jxlpw\" (UID: \"6cb31134-47e9-49b0-9412-e9dbcc5d588a\") " pod="openshift-ingress/kube-auth-proxy-68d884f4c6-jxlpw" Nov 03 20:44:55.077404 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:55.077315 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6cb31134-47e9-49b0-9412-e9dbcc5d588a-tmp\") pod \"kube-auth-proxy-68d884f4c6-jxlpw\" (UID: \"6cb31134-47e9-49b0-9412-e9dbcc5d588a\") " pod="openshift-ingress/kube-auth-proxy-68d884f4c6-jxlpw" Nov 03 20:44:55.080201 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:55.080178 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cb31134-47e9-49b0-9412-e9dbcc5d588a-tls-certs\") pod \"kube-auth-proxy-68d884f4c6-jxlpw\" (UID: \"6cb31134-47e9-49b0-9412-e9dbcc5d588a\") " pod="openshift-ingress/kube-auth-proxy-68d884f4c6-jxlpw" Nov 03 20:44:55.080878 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:55.080858 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6cb31134-47e9-49b0-9412-e9dbcc5d588a-tmp\") pod \"kube-auth-proxy-68d884f4c6-jxlpw\" (UID: \"6cb31134-47e9-49b0-9412-e9dbcc5d588a\") " pod="openshift-ingress/kube-auth-proxy-68d884f4c6-jxlpw" Nov 03 20:44:55.149670 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:55.149635 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjpgl\" (UniqueName: \"kubernetes.io/projected/6cb31134-47e9-49b0-9412-e9dbcc5d588a-kube-api-access-xjpgl\") pod \"kube-auth-proxy-68d884f4c6-jxlpw\" (UID: \"6cb31134-47e9-49b0-9412-e9dbcc5d588a\") " pod="openshift-ingress/kube-auth-proxy-68d884f4c6-jxlpw" Nov 03 20:44:55.190940 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:55.190900 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-68d884f4c6-jxlpw" Nov 03 20:44:55.313552 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:55.313508 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d98hbv2"] Nov 03 20:44:55.319041 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:55.319017 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d98hbv2" Nov 03 20:44:55.376743 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:55.376689 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-qv7jm\"" Nov 03 20:44:55.378889 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:55.378859 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d98hbv2"] Nov 03 20:44:55.483304 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:55.483230 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/621388a7-0f76-444c-b08b-83e93cf26d24-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d98hbv2\" (UID: \"621388a7-0f76-444c-b08b-83e93cf26d24\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d98hbv2" Nov 03 20:44:55.483496 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:55.483301 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/621388a7-0f76-444c-b08b-83e93cf26d24-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d98hbv2\" (UID: \"621388a7-0f76-444c-b08b-83e93cf26d24\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d98hbv2" Nov 03 20:44:55.483496 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:55.483410 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/621388a7-0f76-444c-b08b-83e93cf26d24-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d98hbv2\" (UID: \"621388a7-0f76-444c-b08b-83e93cf26d24\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d98hbv2" Nov 03 20:44:55.483496 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:55.483466 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqfrd\" (UniqueName: \"kubernetes.io/projected/621388a7-0f76-444c-b08b-83e93cf26d24-kube-api-access-fqfrd\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d98hbv2\" (UID: \"621388a7-0f76-444c-b08b-83e93cf26d24\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d98hbv2" Nov 03 20:44:55.483655 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:55.483585 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/621388a7-0f76-444c-b08b-83e93cf26d24-istio-data\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d98hbv2\" (UID: \"621388a7-0f76-444c-b08b-83e93cf26d24\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d98hbv2" Nov 03 20:44:55.483655 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:55.483608 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/621388a7-0f76-444c-b08b-83e93cf26d24-istio-token\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d98hbv2\" (UID: \"621388a7-0f76-444c-b08b-83e93cf26d24\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d98hbv2" Nov 03 20:44:55.483730 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:55.483713 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/621388a7-0f76-444c-b08b-83e93cf26d24-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d98hbv2\" (UID: \"621388a7-0f76-444c-b08b-83e93cf26d24\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d98hbv2" Nov 03 20:44:55.483851 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:55.483831 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/621388a7-0f76-444c-b08b-83e93cf26d24-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d98hbv2\" (UID: \"621388a7-0f76-444c-b08b-83e93cf26d24\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d98hbv2" Nov 03 20:44:55.483911 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:55.483867 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/621388a7-0f76-444c-b08b-83e93cf26d24-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d98hbv2\" (UID: \"621388a7-0f76-444c-b08b-83e93cf26d24\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d98hbv2" Nov 03 20:44:55.578730 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:55.578697 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-68d884f4c6-jxlpw"] Nov 03 20:44:55.581660 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:44:55.581629 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cb31134_47e9_49b0_9412_e9dbcc5d588a.slice/crio-ec83409542976ff571c96383031132fb8ef9b88e6738448684012a91a80277a4 WatchSource:0}: Error finding container ec83409542976ff571c96383031132fb8ef9b88e6738448684012a91a80277a4: Status 404 returned error can't find the container with id ec83409542976ff571c96383031132fb8ef9b88e6738448684012a91a80277a4 Nov 03 20:44:55.584855 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:55.584827 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/621388a7-0f76-444c-b08b-83e93cf26d24-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d98hbv2\" (UID: \"621388a7-0f76-444c-b08b-83e93cf26d24\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d98hbv2" Nov 03 20:44:55.584956 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:55.584888 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/621388a7-0f76-444c-b08b-83e93cf26d24-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d98hbv2\" (UID: \"621388a7-0f76-444c-b08b-83e93cf26d24\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d98hbv2" Nov 03 20:44:55.584956 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:55.584920 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/621388a7-0f76-444c-b08b-83e93cf26d24-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d98hbv2\" (UID: \"621388a7-0f76-444c-b08b-83e93cf26d24\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d98hbv2" Nov 03 20:44:55.584956 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:55.584945 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqfrd\" (UniqueName: \"kubernetes.io/projected/621388a7-0f76-444c-b08b-83e93cf26d24-kube-api-access-fqfrd\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d98hbv2\" (UID: \"621388a7-0f76-444c-b08b-83e93cf26d24\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d98hbv2" Nov 03 20:44:55.585172 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:55.584976 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/621388a7-0f76-444c-b08b-83e93cf26d24-istio-data\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d98hbv2\" (UID: \"621388a7-0f76-444c-b08b-83e93cf26d24\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d98hbv2" Nov 03 20:44:55.585346 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:55.585308 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/621388a7-0f76-444c-b08b-83e93cf26d24-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d98hbv2\" (UID: \"621388a7-0f76-444c-b08b-83e93cf26d24\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d98hbv2" Nov 03 20:44:55.585489 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:55.585354 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/621388a7-0f76-444c-b08b-83e93cf26d24-istio-token\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d98hbv2\" (UID: \"621388a7-0f76-444c-b08b-83e93cf26d24\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d98hbv2" Nov 03 20:44:55.585489 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:55.585406 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/621388a7-0f76-444c-b08b-83e93cf26d24-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d98hbv2\" (UID: \"621388a7-0f76-444c-b08b-83e93cf26d24\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d98hbv2" Nov 03 20:44:55.585610 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:55.585495 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/621388a7-0f76-444c-b08b-83e93cf26d24-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d98hbv2\" (UID: \"621388a7-0f76-444c-b08b-83e93cf26d24\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d98hbv2" Nov 03 20:44:55.585610 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:55.585524 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/621388a7-0f76-444c-b08b-83e93cf26d24-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d98hbv2\" (UID: \"621388a7-0f76-444c-b08b-83e93cf26d24\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d98hbv2" Nov 03 20:44:55.585933 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:55.585640 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/621388a7-0f76-444c-b08b-83e93cf26d24-istio-data\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d98hbv2\" (UID: \"621388a7-0f76-444c-b08b-83e93cf26d24\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d98hbv2" Nov 03 20:44:55.585933 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:55.585819 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/621388a7-0f76-444c-b08b-83e93cf26d24-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d98hbv2\" (UID: \"621388a7-0f76-444c-b08b-83e93cf26d24\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d98hbv2" Nov 03 20:44:55.586049 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:55.585931 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/621388a7-0f76-444c-b08b-83e93cf26d24-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d98hbv2\" (UID: \"621388a7-0f76-444c-b08b-83e93cf26d24\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d98hbv2" Nov 03 20:44:55.586049 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:55.585974 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/621388a7-0f76-444c-b08b-83e93cf26d24-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d98hbv2\" (UID: \"621388a7-0f76-444c-b08b-83e93cf26d24\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d98hbv2" Nov 03 20:44:55.588055 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:55.588030 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/621388a7-0f76-444c-b08b-83e93cf26d24-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d98hbv2\" (UID: \"621388a7-0f76-444c-b08b-83e93cf26d24\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d98hbv2" Nov 03 20:44:55.588229 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:55.588208 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/621388a7-0f76-444c-b08b-83e93cf26d24-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d98hbv2\" (UID: \"621388a7-0f76-444c-b08b-83e93cf26d24\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d98hbv2" Nov 03 20:44:55.646539 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:55.646505 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqfrd\" (UniqueName: \"kubernetes.io/projected/621388a7-0f76-444c-b08b-83e93cf26d24-kube-api-access-fqfrd\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d98hbv2\" (UID: \"621388a7-0f76-444c-b08b-83e93cf26d24\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d98hbv2" Nov 03 20:44:55.646693 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:55.646590 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/621388a7-0f76-444c-b08b-83e93cf26d24-istio-token\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d98hbv2\" (UID: \"621388a7-0f76-444c-b08b-83e93cf26d24\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d98hbv2" Nov 03 20:44:55.935135 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:55.935098 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d98hbv2" Nov 03 20:44:56.223239 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:56.223174 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vhvmb" Nov 03 20:44:56.272827 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:56.272788 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-68d884f4c6-jxlpw" event={"ID":"6cb31134-47e9-49b0-9412-e9dbcc5d588a","Type":"ContainerStarted","Data":"ec83409542976ff571c96383031132fb8ef9b88e6738448684012a91a80277a4"} Nov 03 20:44:56.453456 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:56.453283 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d98hbv2"] Nov 03 20:44:56.463057 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:56.463008 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:44:56.463157 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:56.463111 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:44:56.463198 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:56.463156 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:44:57.280789 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:57.280752 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d98hbv2" event={"ID":"621388a7-0f76-444c-b08b-83e93cf26d24","Type":"ContainerStarted","Data":"731ca1355068cf09f442809820acf0fd45257b0de68b470ef64692c9788c16f7"} Nov 03 20:44:57.281195 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:57.280815 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d98hbv2" event={"ID":"621388a7-0f76-444c-b08b-83e93cf26d24","Type":"ContainerStarted","Data":"0cffc4c87e303d747e12d35fba95f2d59931f70efbbf15ff6328eac54aa4fc40"} Nov 03 20:44:57.935603 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:57.935569 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d98hbv2" Nov 03 20:44:57.941311 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:57.941284 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d98hbv2" Nov 03 20:44:58.075924 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:58.075854 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d98hbv2" podStartSLOduration=3.075800726 podStartE2EDuration="3.075800726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:44:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:44:57.462041097 +0000 UTC m=+332.687360888" watchObservedRunningTime="2026-11-03 20:44:58.075800726 +0000 UTC m=+333.301120517" Nov 03 20:44:58.287567 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:58.287470 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4xr8g" event={"ID":"393b09a4-4340-4396-82fe-f91addfa5568","Type":"ContainerStarted","Data":"485367572bfaa58dec08b9ba226843f86b01cef7c0a48fedf0b60803fe92d877"} Nov 03 20:44:58.288063 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:58.287979 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d98hbv2" Nov 03 20:44:58.289187 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:58.289167 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d98hbv2" Nov 03 20:44:58.406245 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:58.405967 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4xr8g" podStartSLOduration=2.469337949 podStartE2EDuration="8.405946966s" totalImagesPullingTime="5.936609017s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:44:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:44:58.405295645 +0000 UTC m=+333.630615461" watchObservedRunningTime="2026-11-03 20:44:58.405946966 +0000 UTC m=+333.631266757" Nov 03 20:44:59.293677 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:59.293636 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-68d884f4c6-jxlpw" event={"ID":"6cb31134-47e9-49b0-9412-e9dbcc5d588a","Type":"ContainerStarted","Data":"8378ca049d742861475b855124d24bd94f646684efa21d1ac47550c5775925af"} Nov 03 20:44:59.409805 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:44:59.409740 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-68d884f4c6-jxlpw" podStartSLOduration=2.237833427 podStartE2EDuration="5.409720623s" totalImagesPullingTime="3.171887196s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:44:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:44:59.409000181 +0000 UTC m=+334.634319990" watchObservedRunningTime="2026-11-03 20:44:59.409720623 +0000 UTC m=+334.635040414" Nov 03 20:45:13.145238 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:13.145204 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-pn7rj"] Nov 03 20:45:13.149967 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:13.149934 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-pn7rj" Nov 03 20:45:13.204406 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:13.204373 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-pn7rj"] Nov 03 20:45:13.241600 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:13.241570 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlqd6\" (UniqueName: \"kubernetes.io/projected/3d82763b-a4a2-4e48-906f-e30a1b41959c-kube-api-access-wlqd6\") pod \"authorino-57f7f8659f-pn7rj\" (UID: \"3d82763b-a4a2-4e48-906f-e30a1b41959c\") " pod="kuadrant-system/authorino-57f7f8659f-pn7rj" Nov 03 20:45:13.252069 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:13.252040 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-qg6jx\"" Nov 03 20:45:13.342010 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:13.341975 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlqd6\" (UniqueName: \"kubernetes.io/projected/3d82763b-a4a2-4e48-906f-e30a1b41959c-kube-api-access-wlqd6\") pod \"authorino-57f7f8659f-pn7rj\" (UID: \"3d82763b-a4a2-4e48-906f-e30a1b41959c\") " pod="kuadrant-system/authorino-57f7f8659f-pn7rj" Nov 03 20:45:13.402905 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:13.402829 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlqd6\" (UniqueName: \"kubernetes.io/projected/3d82763b-a4a2-4e48-906f-e30a1b41959c-kube-api-access-wlqd6\") pod \"authorino-57f7f8659f-pn7rj\" (UID: \"3d82763b-a4a2-4e48-906f-e30a1b41959c\") " pod="kuadrant-system/authorino-57f7f8659f-pn7rj" Nov 03 20:45:13.463775 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:13.463740 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-pn7rj" Nov 03 20:45:13.831740 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:13.831712 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-pn7rj"] Nov 03 20:45:13.834336 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:45:13.834302 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d82763b_a4a2_4e48_906f_e30a1b41959c.slice/crio-d76fb14de78e415240341c99a22ae81a1dd1529d53f32250736325ad67853dcb WatchSource:0}: Error finding container d76fb14de78e415240341c99a22ae81a1dd1529d53f32250736325ad67853dcb: Status 404 returned error can't find the container with id d76fb14de78e415240341c99a22ae81a1dd1529d53f32250736325ad67853dcb Nov 03 20:45:14.011663 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:14.011630 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vhvmb"] Nov 03 20:45:14.011888 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:14.011867 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vhvmb" podUID="089f2b4a-9ad0-4e71-9efe-a22df561cc8f" containerName="manager" containerID="cri-o://2e6fb409335e3ad40caf77dd10cd717fd08e29ab5ffc6ab024303c4018cecddc" gracePeriod=10 Nov 03 20:45:14.361491 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:14.361428 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-pn7rj" event={"ID":"3d82763b-a4a2-4e48-906f-e30a1b41959c","Type":"ContainerStarted","Data":"d76fb14de78e415240341c99a22ae81a1dd1529d53f32250736325ad67853dcb"} Nov 03 20:45:16.211686 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:16.211616 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vhvmb" Nov 03 20:45:16.271901 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:16.271867 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/089f2b4a-9ad0-4e71-9efe-a22df561cc8f-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/089f2b4a-9ad0-4e71-9efe-a22df561cc8f-extensions-socket-volume\") pod \"089f2b4a-9ad0-4e71-9efe-a22df561cc8f\" (UID: \"089f2b4a-9ad0-4e71-9efe-a22df561cc8f\") " Nov 03 20:45:16.272092 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:16.271957 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/089f2b4a-9ad0-4e71-9efe-a22df561cc8f-kube-api-access-mh8lz\" (UniqueName: \"kubernetes.io/projected/089f2b4a-9ad0-4e71-9efe-a22df561cc8f-kube-api-access-mh8lz\") pod \"089f2b4a-9ad0-4e71-9efe-a22df561cc8f\" (UID: \"089f2b4a-9ad0-4e71-9efe-a22df561cc8f\") " Nov 03 20:45:16.272499 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:16.272464 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/089f2b4a-9ad0-4e71-9efe-a22df561cc8f-extensions-socket-volume" pod "089f2b4a-9ad0-4e71-9efe-a22df561cc8f" (UID: "089f2b4a-9ad0-4e71-9efe-a22df561cc8f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:45:16.274468 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:16.274424 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/089f2b4a-9ad0-4e71-9efe-a22df561cc8f-kube-api-access-mh8lz" pod "089f2b4a-9ad0-4e71-9efe-a22df561cc8f" (UID: "089f2b4a-9ad0-4e71-9efe-a22df561cc8f"). InnerVolumeSpecName "kube-api-access-mh8lz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:45:16.324603 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:45:16.324568 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 20:45:16.372257 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:16.372216 4305 generic.go:356] "Generic (PLEG): container finished" podID="089f2b4a-9ad0-4e71-9efe-a22df561cc8f" containerID="2e6fb409335e3ad40caf77dd10cd717fd08e29ab5ffc6ab024303c4018cecddc" exitCode=0 Nov 03 20:45:16.372465 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:16.372283 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vhvmb" Nov 03 20:45:16.372465 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:16.372349 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vhvmb" event={"ID":"089f2b4a-9ad0-4e71-9efe-a22df561cc8f","Type":"ContainerDied","Data":"2e6fb409335e3ad40caf77dd10cd717fd08e29ab5ffc6ab024303c4018cecddc"} Nov 03 20:45:16.372465 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:16.372389 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vhvmb" event={"ID":"089f2b4a-9ad0-4e71-9efe-a22df561cc8f","Type":"ContainerDied","Data":"56daade9df409a5b09f04df94f0070cf9a46c635c783a2ff317878567adb4e55"} Nov 03 20:45:16.372465 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:16.372412 4305 scope.go:122] "RemoveContainer" containerID="2e6fb409335e3ad40caf77dd10cd717fd08e29ab5ffc6ab024303c4018cecddc" Nov 03 20:45:16.372846 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:16.372817 4305 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/089f2b4a-9ad0-4e71-9efe-a22df561cc8f-extensions-socket-volume\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:45:16.372846 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:16.372848 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mh8lz\" (UniqueName: \"kubernetes.io/projected/089f2b4a-9ad0-4e71-9efe-a22df561cc8f-kube-api-access-mh8lz\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:45:16.383256 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:16.383235 4305 scope.go:122] "RemoveContainer" containerID="2e6fb409335e3ad40caf77dd10cd717fd08e29ab5ffc6ab024303c4018cecddc" Nov 03 20:45:16.383635 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:45:16.383605 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e6fb409335e3ad40caf77dd10cd717fd08e29ab5ffc6ab024303c4018cecddc\": container with ID starting with 2e6fb409335e3ad40caf77dd10cd717fd08e29ab5ffc6ab024303c4018cecddc not found: ID does not exist" containerID="2e6fb409335e3ad40caf77dd10cd717fd08e29ab5ffc6ab024303c4018cecddc" Nov 03 20:45:16.383723 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:16.383652 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e6fb409335e3ad40caf77dd10cd717fd08e29ab5ffc6ab024303c4018cecddc"} err="failed to get container status \"2e6fb409335e3ad40caf77dd10cd717fd08e29ab5ffc6ab024303c4018cecddc\": rpc error: code = NotFound desc = could not find container \"2e6fb409335e3ad40caf77dd10cd717fd08e29ab5ffc6ab024303c4018cecddc\": container with ID starting with 2e6fb409335e3ad40caf77dd10cd717fd08e29ab5ffc6ab024303c4018cecddc not found: ID does not exist" Nov 03 20:45:16.542896 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:16.542856 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vhvmb"] Nov 03 20:45:16.547991 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:16.547963 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vhvmb"] Nov 03 20:45:16.893294 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:16.893266 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="089f2b4a-9ad0-4e71-9efe-a22df561cc8f" path="/var/lib/kubelet/pods/089f2b4a-9ad0-4e71-9efe-a22df561cc8f/volumes" Nov 03 20:45:17.378999 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:17.378918 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-pn7rj" event={"ID":"3d82763b-a4a2-4e48-906f-e30a1b41959c","Type":"ContainerStarted","Data":"5bcc3064f42b42165c184bf3950061c4f921bc2e7933e14d971ab2bc47a85540"} Nov 03 20:45:17.492966 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:17.492886 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-pn7rj" podStartSLOduration=1.8751907060000002 podStartE2EDuration="4.492694561s" totalImagesPullingTime="2.617503855s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:17.492399689 +0000 UTC m=+352.717719479" watchObservedRunningTime="2026-11-03 20:45:17.492694561 +0000 UTC m=+352.718014353" Nov 03 20:45:21.306122 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:21.306086 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-w2wr7"] Nov 03 20:45:21.306543 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:21.306428 4305 cpu_manager.go:414] "containerMap: removing container" podUID="089f2b4a-9ad0-4e71-9efe-a22df561cc8f" containerName="manager" Nov 03 20:45:21.306543 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:21.306454 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="089f2b4a-9ad0-4e71-9efe-a22df561cc8f" containerName="manager" Nov 03 20:45:21.306543 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:21.306509 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="089f2b4a-9ad0-4e71-9efe-a22df561cc8f" containerName="manager" Nov 03 20:45:21.308504 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:21.308485 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-w2wr7" Nov 03 20:45:21.360704 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:21.360671 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 20:45:21.410240 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:21.410210 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 20:45:21.410240 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:21.410210 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-kdp7g\"" Nov 03 20:45:21.418887 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:21.418860 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/67361d70-4a50-4746-a455-ded5894f3807-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-w2wr7\" (UID: \"67361d70-4a50-4746-a455-ded5894f3807\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-w2wr7" Nov 03 20:45:21.419028 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:21.418900 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4pd5\" (UniqueName: \"kubernetes.io/projected/67361d70-4a50-4746-a455-ded5894f3807-kube-api-access-m4pd5\") pod \"kserve-module-controller-manager-7566cf5fc8-w2wr7\" (UID: \"67361d70-4a50-4746-a455-ded5894f3807\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-w2wr7" Nov 03 20:45:21.419028 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:21.418924 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/67361d70-4a50-4746-a455-ded5894f3807-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-w2wr7\" (UID: \"67361d70-4a50-4746-a455-ded5894f3807\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-w2wr7" Nov 03 20:45:21.419110 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:21.419050 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/67361d70-4a50-4746-a455-ded5894f3807-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-w2wr7\" (UID: \"67361d70-4a50-4746-a455-ded5894f3807\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-w2wr7" Nov 03 20:45:21.510040 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:21.510006 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Nov 03 20:45:21.511725 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:21.511701 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-w2wr7"] Nov 03 20:45:21.519695 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:21.519671 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/67361d70-4a50-4746-a455-ded5894f3807-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-w2wr7\" (UID: \"67361d70-4a50-4746-a455-ded5894f3807\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-w2wr7" Nov 03 20:45:21.519789 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:21.519709 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4pd5\" (UniqueName: \"kubernetes.io/projected/67361d70-4a50-4746-a455-ded5894f3807-kube-api-access-m4pd5\") pod \"kserve-module-controller-manager-7566cf5fc8-w2wr7\" (UID: \"67361d70-4a50-4746-a455-ded5894f3807\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-w2wr7" Nov 03 20:45:21.519789 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:21.519731 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/67361d70-4a50-4746-a455-ded5894f3807-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-w2wr7\" (UID: \"67361d70-4a50-4746-a455-ded5894f3807\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-w2wr7" Nov 03 20:45:21.519789 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:21.519766 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/67361d70-4a50-4746-a455-ded5894f3807-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-w2wr7\" (UID: \"67361d70-4a50-4746-a455-ded5894f3807\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-w2wr7" Nov 03 20:45:21.520180 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:21.520164 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/67361d70-4a50-4746-a455-ded5894f3807-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-w2wr7\" (UID: \"67361d70-4a50-4746-a455-ded5894f3807\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-w2wr7" Nov 03 20:45:21.520222 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:21.520193 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/67361d70-4a50-4746-a455-ded5894f3807-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-w2wr7\" (UID: \"67361d70-4a50-4746-a455-ded5894f3807\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-w2wr7" Nov 03 20:45:21.520604 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:21.520589 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/67361d70-4a50-4746-a455-ded5894f3807-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-w2wr7\" (UID: \"67361d70-4a50-4746-a455-ded5894f3807\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-w2wr7" Nov 03 20:45:21.581292 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:21.581218 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4pd5\" (UniqueName: \"kubernetes.io/projected/67361d70-4a50-4746-a455-ded5894f3807-kube-api-access-m4pd5\") pod \"kserve-module-controller-manager-7566cf5fc8-w2wr7\" (UID: \"67361d70-4a50-4746-a455-ded5894f3807\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-w2wr7" Nov 03 20:45:21.619021 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:21.618982 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-w2wr7" Nov 03 20:45:22.040871 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:22.040842 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-w2wr7"] Nov 03 20:45:22.043531 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:45:22.043508 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67361d70_4a50_4746_a455_ded5894f3807.slice/crio-1a0eac0970bd79f3062dbd9f23e68fc9381366d4678fa910cf428f9d7c7b0e4c WatchSource:0}: Error finding container 1a0eac0970bd79f3062dbd9f23e68fc9381366d4678fa910cf428f9d7c7b0e4c: Status 404 returned error can't find the container with id 1a0eac0970bd79f3062dbd9f23e68fc9381366d4678fa910cf428f9d7c7b0e4c Nov 03 20:45:22.400454 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:22.400400 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-w2wr7" event={"ID":"67361d70-4a50-4746-a455-ded5894f3807","Type":"ContainerStarted","Data":"1a0eac0970bd79f3062dbd9f23e68fc9381366d4678fa910cf428f9d7c7b0e4c"} Nov 03 20:45:24.411144 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:24.411111 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-w2wr7" event={"ID":"67361d70-4a50-4746-a455-ded5894f3807","Type":"ContainerStarted","Data":"d5c15871b683c68e918fb4468fe10ae64e080a23afda06a06ffba1c7a3ae3f89"} Nov 03 20:45:24.411561 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:24.411231 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-w2wr7" Nov 03 20:45:24.564241 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:24.564184 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-w2wr7" podStartSLOduration=1.8466086590000002 podStartE2EDuration="3.564168574s" totalImagesPullingTime="1.717559915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:24.562498996 +0000 UTC m=+359.787818804" watchObservedRunningTime="2026-11-03 20:45:24.564168574 +0000 UTC m=+359.789488364" Nov 03 20:45:31.777853 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:31.777817 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 20:45:31.799090 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:31.799058 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-9mqrd" Nov 03 20:45:31.851407 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:31.851378 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 03 20:45:31.902333 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:31.902294 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 20:45:31.915387 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:31.915354 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/58611e0c-31bc-46ca-b60e-a60632c5b273-config-file\") pod \"limitador-limitador-7c9b87d574-9mqrd\" (UID: \"58611e0c-31bc-46ca-b60e-a60632c5b273\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-9mqrd" Nov 03 20:45:31.915387 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:31.915385 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zphkh\" (UniqueName: \"kubernetes.io/projected/58611e0c-31bc-46ca-b60e-a60632c5b273-kube-api-access-zphkh\") pod \"limitador-limitador-7c9b87d574-9mqrd\" (UID: \"58611e0c-31bc-46ca-b60e-a60632c5b273\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-9mqrd" Nov 03 20:45:32.016679 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:32.016640 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/58611e0c-31bc-46ca-b60e-a60632c5b273-config-file\") pod \"limitador-limitador-7c9b87d574-9mqrd\" (UID: \"58611e0c-31bc-46ca-b60e-a60632c5b273\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-9mqrd" Nov 03 20:45:32.016679 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:32.016677 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zphkh\" (UniqueName: \"kubernetes.io/projected/58611e0c-31bc-46ca-b60e-a60632c5b273-kube-api-access-zphkh\") pod \"limitador-limitador-7c9b87d574-9mqrd\" (UID: \"58611e0c-31bc-46ca-b60e-a60632c5b273\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-9mqrd" Nov 03 20:45:32.017282 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:32.017261 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/58611e0c-31bc-46ca-b60e-a60632c5b273-config-file\") pod \"limitador-limitador-7c9b87d574-9mqrd\" (UID: \"58611e0c-31bc-46ca-b60e-a60632c5b273\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-9mqrd" Nov 03 20:45:32.077400 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:32.077337 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zphkh\" (UniqueName: \"kubernetes.io/projected/58611e0c-31bc-46ca-b60e-a60632c5b273-kube-api-access-zphkh\") pod \"limitador-limitador-7c9b87d574-9mqrd\" (UID: \"58611e0c-31bc-46ca-b60e-a60632c5b273\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-9mqrd" Nov 03 20:45:32.096840 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:32.096807 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 20:45:32.111344 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:32.111319 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-9mqrd" Nov 03 20:45:32.494646 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:32.494612 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 20:45:32.497867 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:45:32.497837 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58611e0c_31bc_46ca_b60e_a60632c5b273.slice/crio-986493aa4a36099a10f0b89c29ba19f6a86bd7364bfac70dc0983b29b04de3a6 WatchSource:0}: Error finding container 986493aa4a36099a10f0b89c29ba19f6a86bd7364bfac70dc0983b29b04de3a6: Status 404 returned error can't find the container with id 986493aa4a36099a10f0b89c29ba19f6a86bd7364bfac70dc0983b29b04de3a6 Nov 03 20:45:33.447924 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:33.447889 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-9mqrd" event={"ID":"58611e0c-31bc-46ca-b60e-a60632c5b273","Type":"ContainerStarted","Data":"986493aa4a36099a10f0b89c29ba19f6a86bd7364bfac70dc0983b29b04de3a6"} Nov 03 20:45:35.458589 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:35.458552 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-9mqrd" event={"ID":"58611e0c-31bc-46ca-b60e-a60632c5b273","Type":"ContainerStarted","Data":"5bc1a75e3a9b377fe3b214c2e02ed3e8ce47b7809a5f833dfaf13c376f804582"} Nov 03 20:45:35.459264 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:35.458645 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-9mqrd" Nov 03 20:45:35.587338 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:35.587281 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-9mqrd" podStartSLOduration=2.117174709 podStartE2EDuration="4.587266767s" totalImagesPullingTime="2.470092058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:35.58624531 +0000 UTC m=+370.811565123" watchObservedRunningTime="2026-11-03 20:45:35.587266767 +0000 UTC m=+370.812586556" Nov 03 20:45:36.940120 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:36.940081 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-pn7rj"] Nov 03 20:45:36.940669 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:36.940276 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-pn7rj" podUID="3d82763b-a4a2-4e48-906f-e30a1b41959c" containerName="authorino" containerID="cri-o://5bcc3064f42b42165c184bf3950061c4f921bc2e7933e14d971ab2bc47a85540" gracePeriod=30 Nov 03 20:45:37.200031 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:37.199969 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-pn7rj" Nov 03 20:45:37.270590 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:37.270555 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d82763b-a4a2-4e48-906f-e30a1b41959c-kube-api-access-wlqd6\" (UniqueName: \"kubernetes.io/projected/3d82763b-a4a2-4e48-906f-e30a1b41959c-kube-api-access-wlqd6\") pod \"3d82763b-a4a2-4e48-906f-e30a1b41959c\" (UID: \"3d82763b-a4a2-4e48-906f-e30a1b41959c\") " Nov 03 20:45:37.272792 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:37.272765 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d82763b-a4a2-4e48-906f-e30a1b41959c-kube-api-access-wlqd6" pod "3d82763b-a4a2-4e48-906f-e30a1b41959c" (UID: "3d82763b-a4a2-4e48-906f-e30a1b41959c"). InnerVolumeSpecName "kube-api-access-wlqd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:45:37.371926 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:37.371883 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlqd6\" (UniqueName: \"kubernetes.io/projected/3d82763b-a4a2-4e48-906f-e30a1b41959c-kube-api-access-wlqd6\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:45:37.468380 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:37.468290 4305 generic.go:356] "Generic (PLEG): container finished" podID="3d82763b-a4a2-4e48-906f-e30a1b41959c" containerID="5bcc3064f42b42165c184bf3950061c4f921bc2e7933e14d971ab2bc47a85540" exitCode=0 Nov 03 20:45:37.468380 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:37.468328 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-pn7rj" Nov 03 20:45:37.468380 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:37.468369 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-pn7rj" event={"ID":"3d82763b-a4a2-4e48-906f-e30a1b41959c","Type":"ContainerDied","Data":"5bcc3064f42b42165c184bf3950061c4f921bc2e7933e14d971ab2bc47a85540"} Nov 03 20:45:37.468629 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:37.468412 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-pn7rj" event={"ID":"3d82763b-a4a2-4e48-906f-e30a1b41959c","Type":"ContainerDied","Data":"d76fb14de78e415240341c99a22ae81a1dd1529d53f32250736325ad67853dcb"} Nov 03 20:45:37.468629 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:37.468472 4305 scope.go:122] "RemoveContainer" containerID="5bcc3064f42b42165c184bf3950061c4f921bc2e7933e14d971ab2bc47a85540" Nov 03 20:45:37.477378 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:37.477354 4305 scope.go:122] "RemoveContainer" containerID="5bcc3064f42b42165c184bf3950061c4f921bc2e7933e14d971ab2bc47a85540" Nov 03 20:45:37.477756 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:45:37.477737 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bcc3064f42b42165c184bf3950061c4f921bc2e7933e14d971ab2bc47a85540\": container with ID starting with 5bcc3064f42b42165c184bf3950061c4f921bc2e7933e14d971ab2bc47a85540 not found: ID does not exist" containerID="5bcc3064f42b42165c184bf3950061c4f921bc2e7933e14d971ab2bc47a85540" Nov 03 20:45:37.477816 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:37.477766 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bcc3064f42b42165c184bf3950061c4f921bc2e7933e14d971ab2bc47a85540"} err="failed to get container status \"5bcc3064f42b42165c184bf3950061c4f921bc2e7933e14d971ab2bc47a85540\": rpc error: code = NotFound desc = could not find container \"5bcc3064f42b42165c184bf3950061c4f921bc2e7933e14d971ab2bc47a85540\": container with ID starting with 5bcc3064f42b42165c184bf3950061c4f921bc2e7933e14d971ab2bc47a85540 not found: ID does not exist" Nov 03 20:45:37.633550 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:37.633516 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-pn7rj"] Nov 03 20:45:37.644836 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:37.644812 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-pn7rj"] Nov 03 20:45:38.893663 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:38.893625 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d82763b-a4a2-4e48-906f-e30a1b41959c" path="/var/lib/kubelet/pods/3d82763b-a4a2-4e48-906f-e30a1b41959c/volumes" Nov 03 20:45:45.416899 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:45.416868 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-w2wr7" Nov 03 20:45:46.320451 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:45:46.320400 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 20:45:46.464403 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:45:46.464377 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-9mqrd" Nov 03 20:46:16.320827 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:46:16.320784 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 20:46:28.381835 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:28.381802 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-wmqbg"] Nov 03 20:46:28.382245 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:28.382131 4305 cpu_manager.go:414] "containerMap: removing container" podUID="3d82763b-a4a2-4e48-906f-e30a1b41959c" containerName="authorino" Nov 03 20:46:28.382245 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:28.382141 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d82763b-a4a2-4e48-906f-e30a1b41959c" containerName="authorino" Nov 03 20:46:28.382245 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:28.382213 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d82763b-a4a2-4e48-906f-e30a1b41959c" containerName="authorino" Nov 03 20:46:28.388561 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:28.388535 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-wmqbg" Nov 03 20:46:28.443623 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:28.443589 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-wmqbg"] Nov 03 20:46:28.458579 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:28.458553 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Nov 03 20:46:28.458579 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:28.458558 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-tmck2\"" Nov 03 20:46:28.494210 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:28.494173 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-nmvqg"] Nov 03 20:46:28.497677 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:28.497654 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nmvqg" Nov 03 20:46:28.534674 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:28.534640 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h6lh\" (UniqueName: \"kubernetes.io/projected/b9977cd4-c16f-43e4-a79c-73af81749659-kube-api-access-9h6lh\") pod \"kserve-controller-manager-776cc66d47-wmqbg\" (UID: \"b9977cd4-c16f-43e4-a79c-73af81749659\") " pod="opendatahub/kserve-controller-manager-776cc66d47-wmqbg" Nov 03 20:46:28.534674 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:28.534683 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b9977cd4-c16f-43e4-a79c-73af81749659-cert\") pod \"kserve-controller-manager-776cc66d47-wmqbg\" (UID: \"b9977cd4-c16f-43e4-a79c-73af81749659\") " pod="opendatahub/kserve-controller-manager-776cc66d47-wmqbg" Nov 03 20:46:28.550143 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:28.550114 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-jcv6l\"" Nov 03 20:46:28.550143 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:28.550139 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Nov 03 20:46:28.555557 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:28.555534 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-nmvqg"] Nov 03 20:46:28.636085 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:28.635987 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qwbz\" (UniqueName: \"kubernetes.io/projected/87123303-adac-408d-b273-a8d88351d650-kube-api-access-6qwbz\") pod \"llmisvc-controller-manager-869d5dcc5d-nmvqg\" (UID: \"87123303-adac-408d-b273-a8d88351d650\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nmvqg" Nov 03 20:46:28.636256 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:28.636125 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h6lh\" (UniqueName: \"kubernetes.io/projected/b9977cd4-c16f-43e4-a79c-73af81749659-kube-api-access-9h6lh\") pod \"kserve-controller-manager-776cc66d47-wmqbg\" (UID: \"b9977cd4-c16f-43e4-a79c-73af81749659\") " pod="opendatahub/kserve-controller-manager-776cc66d47-wmqbg" Nov 03 20:46:28.636256 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:28.636164 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/87123303-adac-408d-b273-a8d88351d650-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-nmvqg\" (UID: \"87123303-adac-408d-b273-a8d88351d650\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nmvqg" Nov 03 20:46:28.636256 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:28.636191 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b9977cd4-c16f-43e4-a79c-73af81749659-cert\") pod \"kserve-controller-manager-776cc66d47-wmqbg\" (UID: \"b9977cd4-c16f-43e4-a79c-73af81749659\") " pod="opendatahub/kserve-controller-manager-776cc66d47-wmqbg" Nov 03 20:46:28.638757 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:28.638735 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b9977cd4-c16f-43e4-a79c-73af81749659-cert\") pod \"kserve-controller-manager-776cc66d47-wmqbg\" (UID: \"b9977cd4-c16f-43e4-a79c-73af81749659\") " pod="opendatahub/kserve-controller-manager-776cc66d47-wmqbg" Nov 03 20:46:28.697126 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:28.697095 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h6lh\" (UniqueName: \"kubernetes.io/projected/b9977cd4-c16f-43e4-a79c-73af81749659-kube-api-access-9h6lh\") pod \"kserve-controller-manager-776cc66d47-wmqbg\" (UID: \"b9977cd4-c16f-43e4-a79c-73af81749659\") " pod="opendatahub/kserve-controller-manager-776cc66d47-wmqbg" Nov 03 20:46:28.701877 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:28.701846 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-wmqbg" Nov 03 20:46:28.737450 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:28.737404 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/87123303-adac-408d-b273-a8d88351d650-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-nmvqg\" (UID: \"87123303-adac-408d-b273-a8d88351d650\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nmvqg" Nov 03 20:46:28.737640 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:28.737497 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qwbz\" (UniqueName: \"kubernetes.io/projected/87123303-adac-408d-b273-a8d88351d650-kube-api-access-6qwbz\") pod \"llmisvc-controller-manager-869d5dcc5d-nmvqg\" (UID: \"87123303-adac-408d-b273-a8d88351d650\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nmvqg" Nov 03 20:46:28.739977 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:28.739955 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/87123303-adac-408d-b273-a8d88351d650-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-nmvqg\" (UID: \"87123303-adac-408d-b273-a8d88351d650\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nmvqg" Nov 03 20:46:28.798083 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:28.798054 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qwbz\" (UniqueName: \"kubernetes.io/projected/87123303-adac-408d-b273-a8d88351d650-kube-api-access-6qwbz\") pod \"llmisvc-controller-manager-869d5dcc5d-nmvqg\" (UID: \"87123303-adac-408d-b273-a8d88351d650\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nmvqg" Nov 03 20:46:28.810919 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:28.810890 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nmvqg" Nov 03 20:46:29.075392 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:29.075370 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-wmqbg"] Nov 03 20:46:29.077822 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:46:29.077795 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9977cd4_c16f_43e4_a79c_73af81749659.slice/crio-7012e09f432f3b1b99b493b83eda47bf1ea9638de6d3951626a4f9b6d3d5ab93 WatchSource:0}: Error finding container 7012e09f432f3b1b99b493b83eda47bf1ea9638de6d3951626a4f9b6d3d5ab93: Status 404 returned error can't find the container with id 7012e09f432f3b1b99b493b83eda47bf1ea9638de6d3951626a4f9b6d3d5ab93 Nov 03 20:46:29.182618 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:29.182591 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-nmvqg"] Nov 03 20:46:29.184999 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:46:29.184963 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod87123303_adac_408d_b273_a8d88351d650.slice/crio-22feb0e67a620d8da5dad0892a6c22ebbc495b7c0b2c6f15ba787654420cee11 WatchSource:0}: Error finding container 22feb0e67a620d8da5dad0892a6c22ebbc495b7c0b2c6f15ba787654420cee11: Status 404 returned error can't find the container with id 22feb0e67a620d8da5dad0892a6c22ebbc495b7c0b2c6f15ba787654420cee11 Nov 03 20:46:29.686267 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:29.686235 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nmvqg" event={"ID":"87123303-adac-408d-b273-a8d88351d650","Type":"ContainerStarted","Data":"22feb0e67a620d8da5dad0892a6c22ebbc495b7c0b2c6f15ba787654420cee11"} Nov 03 20:46:29.687343 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:29.687324 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-wmqbg" event={"ID":"b9977cd4-c16f-43e4-a79c-73af81749659","Type":"ContainerStarted","Data":"7012e09f432f3b1b99b493b83eda47bf1ea9638de6d3951626a4f9b6d3d5ab93"} Nov 03 20:46:33.709884 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:33.709843 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nmvqg" event={"ID":"87123303-adac-408d-b273-a8d88351d650","Type":"ContainerStarted","Data":"ac419bd8bf784fb7b271c3e63512f0f685352dd8b76a2d044950623f3e84f8d0"} Nov 03 20:46:33.710411 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:33.710022 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nmvqg" Nov 03 20:46:33.711290 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:33.711265 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-wmqbg" event={"ID":"b9977cd4-c16f-43e4-a79c-73af81749659","Type":"ContainerStarted","Data":"69f5b17efb25d0268e0120664533759a3045a750f323b8d04ae2e152500c2600"} Nov 03 20:46:33.711382 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:33.711323 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-wmqbg" Nov 03 20:46:33.834840 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:33.834791 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nmvqg" podStartSLOduration=2.381443143 podStartE2EDuration="5.834775063s" totalImagesPullingTime="3.45333192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:46:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:46:33.832691388 +0000 UTC m=+429.058011194" watchObservedRunningTime="2026-11-03 20:46:33.834775063 +0000 UTC m=+429.060094853" Nov 03 20:46:33.951681 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:33.951619 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-wmqbg" podStartSLOduration=2.394207104 podStartE2EDuration="5.951600884s" totalImagesPullingTime="3.55739378s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:46:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:46:33.951353663 +0000 UTC m=+429.176673475" watchObservedRunningTime="2026-11-03 20:46:33.951600884 +0000 UTC m=+429.176920673" Nov 03 20:46:41.297592 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:41.297562 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 20:46:46.319972 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:46:46.319939 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 20:46:47.019173 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:47.019133 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-th2tr"] Nov 03 20:46:47.022636 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:47.022613 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-th2tr" Nov 03 20:46:47.075127 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:47.075092 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-qg6jx\"" Nov 03 20:46:47.123620 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:47.123583 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7ldx\" (UniqueName: \"kubernetes.io/projected/8064fef7-2d47-4499-9be7-d721436a9008-kube-api-access-b7ldx\") pod \"authorino-6f6944bcdd-th2tr\" (UID: \"8064fef7-2d47-4499-9be7-d721436a9008\") " pod="kuadrant-system/authorino-6f6944bcdd-th2tr" Nov 03 20:46:47.126333 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:47.126302 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-th2tr"] Nov 03 20:46:47.225277 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:47.225226 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7ldx\" (UniqueName: \"kubernetes.io/projected/8064fef7-2d47-4499-9be7-d721436a9008-kube-api-access-b7ldx\") pod \"authorino-6f6944bcdd-th2tr\" (UID: \"8064fef7-2d47-4499-9be7-d721436a9008\") " pod="kuadrant-system/authorino-6f6944bcdd-th2tr" Nov 03 20:46:47.287393 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:47.287321 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7ldx\" (UniqueName: \"kubernetes.io/projected/8064fef7-2d47-4499-9be7-d721436a9008-kube-api-access-b7ldx\") pod \"authorino-6f6944bcdd-th2tr\" (UID: \"8064fef7-2d47-4499-9be7-d721436a9008\") " pod="kuadrant-system/authorino-6f6944bcdd-th2tr" Nov 03 20:46:47.334374 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:47.334341 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-th2tr" Nov 03 20:46:47.361587 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:47.361554 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-th2tr"] Nov 03 20:46:47.385680 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:47.385607 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5c9df4b67f-t42bs"] Nov 03 20:46:47.391723 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:47.391468 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c9df4b67f-t42bs" Nov 03 20:46:47.428179 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:47.428145 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47wj8\" (UniqueName: \"kubernetes.io/projected/0263828a-fe98-4e72-952a-7dec6cb89fa4-kube-api-access-47wj8\") pod \"authorino-5c9df4b67f-t42bs\" (UID: \"0263828a-fe98-4e72-952a-7dec6cb89fa4\") " pod="kuadrant-system/authorino-5c9df4b67f-t42bs" Nov 03 20:46:47.449649 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:47.449609 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5c9df4b67f-t42bs"] Nov 03 20:46:47.528972 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:47.528938 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47wj8\" (UniqueName: \"kubernetes.io/projected/0263828a-fe98-4e72-952a-7dec6cb89fa4-kube-api-access-47wj8\") pod \"authorino-5c9df4b67f-t42bs\" (UID: \"0263828a-fe98-4e72-952a-7dec6cb89fa4\") " pod="kuadrant-system/authorino-5c9df4b67f-t42bs" Nov 03 20:46:47.590348 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:47.590278 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47wj8\" (UniqueName: \"kubernetes.io/projected/0263828a-fe98-4e72-952a-7dec6cb89fa4-kube-api-access-47wj8\") pod \"authorino-5c9df4b67f-t42bs\" (UID: \"0263828a-fe98-4e72-952a-7dec6cb89fa4\") " pod="kuadrant-system/authorino-5c9df4b67f-t42bs" Nov 03 20:46:47.707697 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:47.707665 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-th2tr"] Nov 03 20:46:47.711756 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:47.711733 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c9df4b67f-t42bs" Nov 03 20:46:47.711922 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:46:47.711850 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8064fef7_2d47_4499_9be7_d721436a9008.slice/crio-f217364deb137531a4eec3323b8adaaba8c6ab422a651d874d85b5eca8ef8cc4 WatchSource:0}: Error finding container f217364deb137531a4eec3323b8adaaba8c6ab422a651d874d85b5eca8ef8cc4: Status 404 returned error can't find the container with id f217364deb137531a4eec3323b8adaaba8c6ab422a651d874d85b5eca8ef8cc4 Nov 03 20:46:47.798504 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:47.798472 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-th2tr" event={"ID":"8064fef7-2d47-4499-9be7-d721436a9008","Type":"ContainerStarted","Data":"f217364deb137531a4eec3323b8adaaba8c6ab422a651d874d85b5eca8ef8cc4"} Nov 03 20:46:47.846308 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:47.846231 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5c9df4b67f-t42bs"] Nov 03 20:46:47.871890 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:47.871828 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-857bc5d6df-rncbs"] Nov 03 20:46:47.875399 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:47.875369 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-857bc5d6df-rncbs" Nov 03 20:46:47.930027 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:47.929988 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 03 20:46:47.932243 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:47.932217 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-857bc5d6df-rncbs"] Nov 03 20:46:47.932589 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:47.932568 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a060940f-3a28-4af9-9fa0-724b4ace3a2e-tls-cert\") pod \"authorino-857bc5d6df-rncbs\" (UID: \"a060940f-3a28-4af9-9fa0-724b4ace3a2e\") " pod="kuadrant-system/authorino-857bc5d6df-rncbs" Nov 03 20:46:47.932736 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:47.932721 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk5jp\" (UniqueName: \"kubernetes.io/projected/a060940f-3a28-4af9-9fa0-724b4ace3a2e-kube-api-access-nk5jp\") pod \"authorino-857bc5d6df-rncbs\" (UID: \"a060940f-3a28-4af9-9fa0-724b4ace3a2e\") " pod="kuadrant-system/authorino-857bc5d6df-rncbs" Nov 03 20:46:48.033607 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:48.033572 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk5jp\" (UniqueName: \"kubernetes.io/projected/a060940f-3a28-4af9-9fa0-724b4ace3a2e-kube-api-access-nk5jp\") pod \"authorino-857bc5d6df-rncbs\" (UID: \"a060940f-3a28-4af9-9fa0-724b4ace3a2e\") " pod="kuadrant-system/authorino-857bc5d6df-rncbs" Nov 03 20:46:48.033799 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:48.033629 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a060940f-3a28-4af9-9fa0-724b4ace3a2e-tls-cert\") pod \"authorino-857bc5d6df-rncbs\" (UID: \"a060940f-3a28-4af9-9fa0-724b4ace3a2e\") " pod="kuadrant-system/authorino-857bc5d6df-rncbs" Nov 03 20:46:48.036382 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:48.036348 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a060940f-3a28-4af9-9fa0-724b4ace3a2e-tls-cert\") pod \"authorino-857bc5d6df-rncbs\" (UID: \"a060940f-3a28-4af9-9fa0-724b4ace3a2e\") " pod="kuadrant-system/authorino-857bc5d6df-rncbs" Nov 03 20:46:48.090276 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:48.090254 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5c9df4b67f-t42bs"] Nov 03 20:46:48.092998 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:46:48.092962 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0263828a_fe98_4e72_952a_7dec6cb89fa4.slice/crio-08c66a789c82d0913dcda36411ce9f9ce17c6edc803265a3d102ae42d4c3020f WatchSource:0}: Error finding container 08c66a789c82d0913dcda36411ce9f9ce17c6edc803265a3d102ae42d4c3020f: Status 404 returned error can't find the container with id 08c66a789c82d0913dcda36411ce9f9ce17c6edc803265a3d102ae42d4c3020f Nov 03 20:46:48.096495 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:48.096422 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk5jp\" (UniqueName: \"kubernetes.io/projected/a060940f-3a28-4af9-9fa0-724b4ace3a2e-kube-api-access-nk5jp\") pod \"authorino-857bc5d6df-rncbs\" (UID: \"a060940f-3a28-4af9-9fa0-724b4ace3a2e\") " pod="kuadrant-system/authorino-857bc5d6df-rncbs" Nov 03 20:46:48.188522 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:48.188479 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-857bc5d6df-rncbs" Nov 03 20:46:48.561392 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:48.561359 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-857bc5d6df-rncbs"] Nov 03 20:46:48.565038 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:46:48.565012 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda060940f_3a28_4af9_9fa0_724b4ace3a2e.slice/crio-07b8e875ded65ac99cc2b534e51e9e2f40272f154c254e14edd3c14ab300b3f0 WatchSource:0}: Error finding container 07b8e875ded65ac99cc2b534e51e9e2f40272f154c254e14edd3c14ab300b3f0: Status 404 returned error can't find the container with id 07b8e875ded65ac99cc2b534e51e9e2f40272f154c254e14edd3c14ab300b3f0 Nov 03 20:46:48.805494 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:48.805462 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-857bc5d6df-rncbs" event={"ID":"a060940f-3a28-4af9-9fa0-724b4ace3a2e","Type":"ContainerStarted","Data":"07b8e875ded65ac99cc2b534e51e9e2f40272f154c254e14edd3c14ab300b3f0"} Nov 03 20:46:48.806661 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:48.806642 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c9df4b67f-t42bs" event={"ID":"0263828a-fe98-4e72-952a-7dec6cb89fa4","Type":"ContainerStarted","Data":"08c66a789c82d0913dcda36411ce9f9ce17c6edc803265a3d102ae42d4c3020f"} Nov 03 20:46:49.814652 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:49.814556 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c9df4b67f-t42bs" event={"ID":"0263828a-fe98-4e72-952a-7dec6cb89fa4","Type":"ContainerStarted","Data":"4ab99439e0f0c67361a0a3ff3611c18adeb91c4b6d07d5356676d195e5fa8195"} Nov 03 20:46:49.814652 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:49.814592 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-5c9df4b67f-t42bs" podUID="0263828a-fe98-4e72-952a-7dec6cb89fa4" containerName="authorino" containerID="cri-o://4ab99439e0f0c67361a0a3ff3611c18adeb91c4b6d07d5356676d195e5fa8195" gracePeriod=30 Nov 03 20:46:49.816874 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:49.816845 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-th2tr" event={"ID":"8064fef7-2d47-4499-9be7-d721436a9008","Type":"ContainerStarted","Data":"3f6bf682c038dcd492cca7fadebfcfa40b54354f1b2c9e762b2bf95f03e2258d"} Nov 03 20:46:49.816997 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:49.816922 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-th2tr" podUID="8064fef7-2d47-4499-9be7-d721436a9008" containerName="authorino" containerID="cri-o://3f6bf682c038dcd492cca7fadebfcfa40b54354f1b2c9e762b2bf95f03e2258d" gracePeriod=30 Nov 03 20:46:49.929873 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:49.929779 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5c9df4b67f-t42bs" podStartSLOduration=1.554220776 podStartE2EDuration="2.929762753s" totalImagesPullingTime="1.375541977s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:46:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:46:49.92757105 +0000 UTC m=+445.152890842" watchObservedRunningTime="2026-11-03 20:46:49.929762753 +0000 UTC m=+445.155082544" Nov 03 20:46:50.078025 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:50.077989 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c9df4b67f-t42bs" Nov 03 20:46:50.116320 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:50.116291 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-th2tr" Nov 03 20:46:50.152360 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:50.152327 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8064fef7-2d47-4499-9be7-d721436a9008-kube-api-access-b7ldx\" (UniqueName: \"kubernetes.io/projected/8064fef7-2d47-4499-9be7-d721436a9008-kube-api-access-b7ldx\") pod \"8064fef7-2d47-4499-9be7-d721436a9008\" (UID: \"8064fef7-2d47-4499-9be7-d721436a9008\") " Nov 03 20:46:50.152575 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:50.152384 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0263828a-fe98-4e72-952a-7dec6cb89fa4-kube-api-access-47wj8\" (UniqueName: \"kubernetes.io/projected/0263828a-fe98-4e72-952a-7dec6cb89fa4-kube-api-access-47wj8\") pod \"0263828a-fe98-4e72-952a-7dec6cb89fa4\" (UID: \"0263828a-fe98-4e72-952a-7dec6cb89fa4\") " Nov 03 20:46:50.154898 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:50.154864 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8064fef7-2d47-4499-9be7-d721436a9008-kube-api-access-b7ldx" pod "8064fef7-2d47-4499-9be7-d721436a9008" (UID: "8064fef7-2d47-4499-9be7-d721436a9008"). InnerVolumeSpecName "kube-api-access-b7ldx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:46:50.155013 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:50.154963 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0263828a-fe98-4e72-952a-7dec6cb89fa4-kube-api-access-47wj8" pod "0263828a-fe98-4e72-952a-7dec6cb89fa4" (UID: "0263828a-fe98-4e72-952a-7dec6cb89fa4"). InnerVolumeSpecName "kube-api-access-47wj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:46:50.192640 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:50.192582 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-th2tr" podStartSLOduration=2.680813272 podStartE2EDuration="4.192563608s" totalImagesPullingTime="1.511750336s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:46:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:46:50.044403141 +0000 UTC m=+445.269722958" watchObservedRunningTime="2026-11-03 20:46:50.192563608 +0000 UTC m=+445.417883399" Nov 03 20:46:50.253650 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:50.253615 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7ldx\" (UniqueName: \"kubernetes.io/projected/8064fef7-2d47-4499-9be7-d721436a9008-kube-api-access-b7ldx\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:46:50.253650 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:50.253651 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47wj8\" (UniqueName: \"kubernetes.io/projected/0263828a-fe98-4e72-952a-7dec6cb89fa4-kube-api-access-47wj8\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:46:50.825070 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:50.824761 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-857bc5d6df-rncbs" event={"ID":"a060940f-3a28-4af9-9fa0-724b4ace3a2e","Type":"ContainerStarted","Data":"777c0ee4dfb33bfe38d61acd5c0c2e89263436014db2f895762a6153898a0311"} Nov 03 20:46:50.827109 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:50.827082 4305 generic.go:356] "Generic (PLEG): container finished" podID="c97844f3-937c-4873-865a-c408d83b0890" containerID="71272b9187b8c275851e3b0e36a1ba0cbbf1f181207413e4bf02904c4edd0d07" exitCode=0 Nov 03 20:46:50.827261 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:50.827154 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd" event={"ID":"c97844f3-937c-4873-865a-c408d83b0890","Type":"ContainerDied","Data":"71272b9187b8c275851e3b0e36a1ba0cbbf1f181207413e4bf02904c4edd0d07"} Nov 03 20:46:50.832497 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:50.827500 4305 scope.go:122] "RemoveContainer" containerID="71272b9187b8c275851e3b0e36a1ba0cbbf1f181207413e4bf02904c4edd0d07" Nov 03 20:46:50.833552 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:50.833135 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:46:50.833552 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:50.833224 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:46:50.833552 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:50.833268 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:46:50.834420 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:50.834397 4305 generic.go:356] "Generic (PLEG): container finished" podID="8064fef7-2d47-4499-9be7-d721436a9008" containerID="3f6bf682c038dcd492cca7fadebfcfa40b54354f1b2c9e762b2bf95f03e2258d" exitCode=0 Nov 03 20:46:50.834583 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:50.834515 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-th2tr" Nov 03 20:46:50.834583 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:50.834558 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-th2tr" event={"ID":"8064fef7-2d47-4499-9be7-d721436a9008","Type":"ContainerDied","Data":"3f6bf682c038dcd492cca7fadebfcfa40b54354f1b2c9e762b2bf95f03e2258d"} Nov 03 20:46:50.834699 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:50.834602 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-th2tr" event={"ID":"8064fef7-2d47-4499-9be7-d721436a9008","Type":"ContainerDied","Data":"f217364deb137531a4eec3323b8adaaba8c6ab422a651d874d85b5eca8ef8cc4"} Nov 03 20:46:50.834699 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:50.834626 4305 scope.go:122] "RemoveContainer" containerID="3f6bf682c038dcd492cca7fadebfcfa40b54354f1b2c9e762b2bf95f03e2258d" Nov 03 20:46:50.841005 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:50.840968 4305 generic.go:356] "Generic (PLEG): container finished" podID="0263828a-fe98-4e72-952a-7dec6cb89fa4" containerID="4ab99439e0f0c67361a0a3ff3611c18adeb91c4b6d07d5356676d195e5fa8195" exitCode=0 Nov 03 20:46:50.841126 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:50.841035 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c9df4b67f-t42bs" event={"ID":"0263828a-fe98-4e72-952a-7dec6cb89fa4","Type":"ContainerDied","Data":"4ab99439e0f0c67361a0a3ff3611c18adeb91c4b6d07d5356676d195e5fa8195"} Nov 03 20:46:50.841126 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:50.841062 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c9df4b67f-t42bs" event={"ID":"0263828a-fe98-4e72-952a-7dec6cb89fa4","Type":"ContainerDied","Data":"08c66a789c82d0913dcda36411ce9f9ce17c6edc803265a3d102ae42d4c3020f"} Nov 03 20:46:50.841126 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:50.841115 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c9df4b67f-t42bs" Nov 03 20:46:50.850496 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:50.850467 4305 scope.go:122] "RemoveContainer" containerID="3f6bf682c038dcd492cca7fadebfcfa40b54354f1b2c9e762b2bf95f03e2258d" Nov 03 20:46:50.851015 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:46:50.850996 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f6bf682c038dcd492cca7fadebfcfa40b54354f1b2c9e762b2bf95f03e2258d\": container with ID starting with 3f6bf682c038dcd492cca7fadebfcfa40b54354f1b2c9e762b2bf95f03e2258d not found: ID does not exist" containerID="3f6bf682c038dcd492cca7fadebfcfa40b54354f1b2c9e762b2bf95f03e2258d" Nov 03 20:46:50.851112 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:50.851029 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f6bf682c038dcd492cca7fadebfcfa40b54354f1b2c9e762b2bf95f03e2258d"} err="failed to get container status \"3f6bf682c038dcd492cca7fadebfcfa40b54354f1b2c9e762b2bf95f03e2258d\": rpc error: code = NotFound desc = could not find container \"3f6bf682c038dcd492cca7fadebfcfa40b54354f1b2c9e762b2bf95f03e2258d\": container with ID starting with 3f6bf682c038dcd492cca7fadebfcfa40b54354f1b2c9e762b2bf95f03e2258d not found: ID does not exist" Nov 03 20:46:50.851112 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:50.851053 4305 scope.go:122] "RemoveContainer" containerID="4ab99439e0f0c67361a0a3ff3611c18adeb91c4b6d07d5356676d195e5fa8195" Nov 03 20:46:50.860775 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:50.860753 4305 scope.go:122] "RemoveContainer" containerID="4ab99439e0f0c67361a0a3ff3611c18adeb91c4b6d07d5356676d195e5fa8195" Nov 03 20:46:50.861081 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:46:50.861065 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ab99439e0f0c67361a0a3ff3611c18adeb91c4b6d07d5356676d195e5fa8195\": container with ID starting with 4ab99439e0f0c67361a0a3ff3611c18adeb91c4b6d07d5356676d195e5fa8195 not found: ID does not exist" containerID="4ab99439e0f0c67361a0a3ff3611c18adeb91c4b6d07d5356676d195e5fa8195" Nov 03 20:46:50.861145 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:50.861089 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ab99439e0f0c67361a0a3ff3611c18adeb91c4b6d07d5356676d195e5fa8195"} err="failed to get container status \"4ab99439e0f0c67361a0a3ff3611c18adeb91c4b6d07d5356676d195e5fa8195\": rpc error: code = NotFound desc = could not find container \"4ab99439e0f0c67361a0a3ff3611c18adeb91c4b6d07d5356676d195e5fa8195\": container with ID starting with 4ab99439e0f0c67361a0a3ff3611c18adeb91c4b6d07d5356676d195e5fa8195 not found: ID does not exist" Nov 03 20:46:50.918339 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:50.918302 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd" Nov 03 20:46:50.939092 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:50.939030 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-857bc5d6df-rncbs" podStartSLOduration=2.690207557 podStartE2EDuration="3.939012821s" totalImagesPullingTime="1.248805264s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:46:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:46:50.938592318 +0000 UTC m=+446.163912113" watchObservedRunningTime="2026-11-03 20:46:50.939012821 +0000 UTC m=+446.164332612" Nov 03 20:46:51.053092 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:51.053061 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-th2tr"] Nov 03 20:46:51.057468 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:51.057420 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-th2tr"] Nov 03 20:46:51.285323 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:51.285293 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5c9df4b67f-t42bs"] Nov 03 20:46:51.288197 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:51.288165 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5c9df4b67f-t42bs"] Nov 03 20:46:51.847736 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:51.847705 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd" event={"ID":"c97844f3-937c-4873-865a-c408d83b0890","Type":"ContainerStarted","Data":"3c92e1c0e4ef243987942b115cada19a70ea9c7b130d49dc0a0f47344c6987bd"} Nov 03 20:46:52.742712 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:52.742676 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd" Nov 03 20:46:52.743955 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:52.743932 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body= Nov 03 20:46:52.744089 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:52.743993 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd" podUID="c97844f3-937c-4873-865a-c408d83b0890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" Nov 03 20:46:52.894109 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:52.894073 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0263828a-fe98-4e72-952a-7dec6cb89fa4" path="/var/lib/kubelet/pods/0263828a-fe98-4e72-952a-7dec6cb89fa4/volumes" Nov 03 20:46:52.894601 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:52.894393 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8064fef7-2d47-4499-9be7-d721436a9008" path="/var/lib/kubelet/pods/8064fef7-2d47-4499-9be7-d721436a9008/volumes" Nov 03 20:46:53.742506 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:53.742466 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body= Nov 03 20:46:53.742713 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:53.742542 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd" podUID="c97844f3-937c-4873-865a-c408d83b0890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" Nov 03 20:46:54.742371 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:54.742334 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body= Nov 03 20:46:54.742796 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:54.742417 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd" podUID="c97844f3-937c-4873-865a-c408d83b0890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" Nov 03 20:46:55.742875 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:55.742839 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body= Nov 03 20:46:55.743271 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:55.742893 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd" podUID="c97844f3-937c-4873-865a-c408d83b0890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" Nov 03 20:46:56.742540 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:56.742500 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body= Nov 03 20:46:56.742795 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:56.742563 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd" podUID="c97844f3-937c-4873-865a-c408d83b0890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" Nov 03 20:46:57.746177 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:57.746147 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd" Nov 03 20:46:57.746580 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:57.746514 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd" Nov 03 20:46:57.747247 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:46:57.747233 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd" Nov 03 20:47:04.717827 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:47:04.717798 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nmvqg" Nov 03 20:47:04.720661 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:47:04.720638 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-wmqbg" Nov 03 20:47:16.319670 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:47:16.319630 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 20:47:46.320545 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:47:46.320454 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 20:48:16.322128 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:48:16.322085 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 20:48:27.665202 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:48:27.665171 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-nmvqg"] Nov 03 20:48:27.665683 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:48:27.665377 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nmvqg" podUID="87123303-adac-408d-b273-a8d88351d650" containerName="manager" containerID="cri-o://ac419bd8bf784fb7b271c3e63512f0f685352dd8b76a2d044950623f3e84f8d0" gracePeriod=30 Nov 03 20:48:27.910865 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:48:27.910840 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nmvqg" Nov 03 20:48:27.951228 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:48:27.951139 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87123303-adac-408d-b273-a8d88351d650-kube-api-access-6qwbz\" (UniqueName: \"kubernetes.io/projected/87123303-adac-408d-b273-a8d88351d650-kube-api-access-6qwbz\") pod \"87123303-adac-408d-b273-a8d88351d650\" (UID: \"87123303-adac-408d-b273-a8d88351d650\") " Nov 03 20:48:27.951366 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:48:27.951245 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87123303-adac-408d-b273-a8d88351d650-cert\" (UniqueName: \"kubernetes.io/secret/87123303-adac-408d-b273-a8d88351d650-cert\") pod \"87123303-adac-408d-b273-a8d88351d650\" (UID: \"87123303-adac-408d-b273-a8d88351d650\") " Nov 03 20:48:27.953581 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:48:27.953551 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87123303-adac-408d-b273-a8d88351d650-kube-api-access-6qwbz" pod "87123303-adac-408d-b273-a8d88351d650" (UID: "87123303-adac-408d-b273-a8d88351d650"). InnerVolumeSpecName "kube-api-access-6qwbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:48:27.953693 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:48:27.953599 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87123303-adac-408d-b273-a8d88351d650-cert" pod "87123303-adac-408d-b273-a8d88351d650" (UID: "87123303-adac-408d-b273-a8d88351d650"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:48:28.052721 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:48:28.052672 4305 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/87123303-adac-408d-b273-a8d88351d650-cert\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:48:28.052721 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:48:28.052725 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qwbz\" (UniqueName: \"kubernetes.io/projected/87123303-adac-408d-b273-a8d88351d650-kube-api-access-6qwbz\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:48:28.292752 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:48:28.292666 4305 generic.go:356] "Generic (PLEG): container finished" podID="87123303-adac-408d-b273-a8d88351d650" containerID="ac419bd8bf784fb7b271c3e63512f0f685352dd8b76a2d044950623f3e84f8d0" exitCode=0 Nov 03 20:48:28.292752 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:48:28.292709 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nmvqg" Nov 03 20:48:28.293004 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:48:28.292746 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nmvqg" event={"ID":"87123303-adac-408d-b273-a8d88351d650","Type":"ContainerDied","Data":"ac419bd8bf784fb7b271c3e63512f0f685352dd8b76a2d044950623f3e84f8d0"} Nov 03 20:48:28.293004 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:48:28.292829 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nmvqg" event={"ID":"87123303-adac-408d-b273-a8d88351d650","Type":"ContainerDied","Data":"22feb0e67a620d8da5dad0892a6c22ebbc495b7c0b2c6f15ba787654420cee11"} Nov 03 20:48:28.293004 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:48:28.292861 4305 scope.go:122] "RemoveContainer" containerID="ac419bd8bf784fb7b271c3e63512f0f685352dd8b76a2d044950623f3e84f8d0" Nov 03 20:48:28.301382 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:48:28.301358 4305 scope.go:122] "RemoveContainer" containerID="ac419bd8bf784fb7b271c3e63512f0f685352dd8b76a2d044950623f3e84f8d0" Nov 03 20:48:28.301689 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:48:28.301669 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac419bd8bf784fb7b271c3e63512f0f685352dd8b76a2d044950623f3e84f8d0\": container with ID starting with ac419bd8bf784fb7b271c3e63512f0f685352dd8b76a2d044950623f3e84f8d0 not found: ID does not exist" containerID="ac419bd8bf784fb7b271c3e63512f0f685352dd8b76a2d044950623f3e84f8d0" Nov 03 20:48:28.301742 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:48:28.301698 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac419bd8bf784fb7b271c3e63512f0f685352dd8b76a2d044950623f3e84f8d0"} err="failed to get container status \"ac419bd8bf784fb7b271c3e63512f0f685352dd8b76a2d044950623f3e84f8d0\": rpc error: code = NotFound desc = could not find container \"ac419bd8bf784fb7b271c3e63512f0f685352dd8b76a2d044950623f3e84f8d0\": container with ID starting with ac419bd8bf784fb7b271c3e63512f0f685352dd8b76a2d044950623f3e84f8d0 not found: ID does not exist" Nov 03 20:48:28.458728 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:48:28.458695 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-nmvqg"] Nov 03 20:48:28.463297 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:48:28.463274 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-nmvqg"] Nov 03 20:48:28.893798 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:48:28.893768 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87123303-adac-408d-b273-a8d88351d650" path="/var/lib/kubelet/pods/87123303-adac-408d-b273-a8d88351d650/volumes" Nov 03 20:48:46.320026 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:48:46.319987 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 20:48:49.596177 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:48:49.596139 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-c9n6g"] Nov 03 20:48:49.596640 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:48:49.596517 4305 cpu_manager.go:414] "containerMap: removing container" podUID="87123303-adac-408d-b273-a8d88351d650" containerName="manager" Nov 03 20:48:49.596640 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:48:49.596529 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87123303-adac-408d-b273-a8d88351d650" containerName="manager" Nov 03 20:48:49.596640 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:48:49.596553 4305 cpu_manager.go:414] "containerMap: removing container" podUID="8064fef7-2d47-4499-9be7-d721436a9008" containerName="authorino" Nov 03 20:48:49.596640 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:48:49.596558 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8064fef7-2d47-4499-9be7-d721436a9008" containerName="authorino" Nov 03 20:48:49.596640 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:48:49.596573 4305 cpu_manager.go:414] "containerMap: removing container" podUID="0263828a-fe98-4e72-952a-7dec6cb89fa4" containerName="authorino" Nov 03 20:48:49.596640 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:48:49.596578 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0263828a-fe98-4e72-952a-7dec6cb89fa4" containerName="authorino" Nov 03 20:48:49.596640 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:48:49.596636 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="87123303-adac-408d-b273-a8d88351d650" containerName="manager" Nov 03 20:48:49.596640 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:48:49.596646 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="8064fef7-2d47-4499-9be7-d721436a9008" containerName="authorino" Nov 03 20:48:49.596925 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:48:49.596654 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="0263828a-fe98-4e72-952a-7dec6cb89fa4" containerName="authorino" Nov 03 20:48:49.599717 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:48:49.599702 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c9n6g" Nov 03 20:48:49.652014 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:48:49.651978 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 03 20:48:49.701543 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:48:49.701508 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-wnljc\"" Nov 03 20:48:49.701543 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:48:49.701525 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 03 20:48:49.701763 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:48:49.701565 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Nov 03 20:48:49.703095 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:48:49.703075 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-c9n6g"] Nov 03 20:48:49.752237 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:48:49.752202 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e264222-b7cf-4f08-85ff-f274be9562b5-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c9n6g\" (UID: \"5e264222-b7cf-4f08-85ff-f274be9562b5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c9n6g" Nov 03 20:48:49.752404 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:48:49.752266 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e264222-b7cf-4f08-85ff-f274be9562b5-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c9n6g\" (UID: \"5e264222-b7cf-4f08-85ff-f274be9562b5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c9n6g" Nov 03 20:48:49.752404 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:48:49.752365 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e264222-b7cf-4f08-85ff-f274be9562b5-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c9n6g\" (UID: \"5e264222-b7cf-4f08-85ff-f274be9562b5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c9n6g" Nov 03 20:48:49.752639 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:48:49.752406 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e264222-b7cf-4f08-85ff-f274be9562b5-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c9n6g\" (UID: \"5e264222-b7cf-4f08-85ff-f274be9562b5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c9n6g" Nov 03 20:48:49.752639 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:48:49.752424 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqtml\" (UniqueName: \"kubernetes.io/projected/5e264222-b7cf-4f08-85ff-f274be9562b5-kube-api-access-nqtml\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c9n6g\" (UID: \"5e264222-b7cf-4f08-85ff-f274be9562b5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c9n6g" Nov 03 20:48:49.752639 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:48:49.752580 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e264222-b7cf-4f08-85ff-f274be9562b5-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c9n6g\" (UID: \"5e264222-b7cf-4f08-85ff-f274be9562b5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c9n6g" Nov 03 20:48:49.853856 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:48:49.853772 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e264222-b7cf-4f08-85ff-f274be9562b5-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c9n6g\" (UID: \"5e264222-b7cf-4f08-85ff-f274be9562b5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c9n6g" Nov 03 20:48:49.853856 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:48:49.853813 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e264222-b7cf-4f08-85ff-f274be9562b5-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c9n6g\" (UID: \"5e264222-b7cf-4f08-85ff-f274be9562b5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c9n6g" Nov 03 20:48:49.853856 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:48:49.853845 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e264222-b7cf-4f08-85ff-f274be9562b5-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c9n6g\" (UID: \"5e264222-b7cf-4f08-85ff-f274be9562b5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c9n6g" Nov 03 20:48:49.854144 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:48:49.853867 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqtml\" (UniqueName: \"kubernetes.io/projected/5e264222-b7cf-4f08-85ff-f274be9562b5-kube-api-access-nqtml\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c9n6g\" (UID: \"5e264222-b7cf-4f08-85ff-f274be9562b5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c9n6g" Nov 03 20:48:49.854144 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:48:49.853917 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e264222-b7cf-4f08-85ff-f274be9562b5-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c9n6g\" (UID: \"5e264222-b7cf-4f08-85ff-f274be9562b5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c9n6g" Nov 03 20:48:49.854144 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:48:49.853975 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e264222-b7cf-4f08-85ff-f274be9562b5-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c9n6g\" (UID: \"5e264222-b7cf-4f08-85ff-f274be9562b5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c9n6g" Nov 03 20:48:49.854295 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:48:49.854262 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e264222-b7cf-4f08-85ff-f274be9562b5-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c9n6g\" (UID: \"5e264222-b7cf-4f08-85ff-f274be9562b5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c9n6g" Nov 03 20:48:49.854348 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:48:49.854317 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e264222-b7cf-4f08-85ff-f274be9562b5-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c9n6g\" (UID: \"5e264222-b7cf-4f08-85ff-f274be9562b5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c9n6g" Nov 03 20:48:49.854386 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:48:49.854338 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e264222-b7cf-4f08-85ff-f274be9562b5-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c9n6g\" (UID: \"5e264222-b7cf-4f08-85ff-f274be9562b5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c9n6g" Nov 03 20:48:49.856239 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:48:49.856215 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e264222-b7cf-4f08-85ff-f274be9562b5-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c9n6g\" (UID: \"5e264222-b7cf-4f08-85ff-f274be9562b5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c9n6g" Nov 03 20:48:49.856395 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:48:49.856377 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e264222-b7cf-4f08-85ff-f274be9562b5-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c9n6g\" (UID: \"5e264222-b7cf-4f08-85ff-f274be9562b5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c9n6g" Nov 03 20:48:49.915598 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:48:49.915564 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqtml\" (UniqueName: \"kubernetes.io/projected/5e264222-b7cf-4f08-85ff-f274be9562b5-kube-api-access-nqtml\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c9n6g\" (UID: \"5e264222-b7cf-4f08-85ff-f274be9562b5\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c9n6g" Nov 03 20:48:50.212588 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:48:50.212549 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c9n6g" Nov 03 20:48:50.583767 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:48:50.583737 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-c9n6g"] Nov 03 20:48:50.588713 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:48:50.586268 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e264222_b7cf_4f08_85ff_f274be9562b5.slice/crio-dd2ca1801b54deace4aa6aac1cd1f75d5cc2972de64e95425b10bb8b14719ace WatchSource:0}: Error finding container dd2ca1801b54deace4aa6aac1cd1f75d5cc2972de64e95425b10bb8b14719ace: Status 404 returned error can't find the container with id dd2ca1801b54deace4aa6aac1cd1f75d5cc2972de64e95425b10bb8b14719ace Nov 03 20:48:51.393112 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:48:51.393083 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c9n6g" event={"ID":"5e264222-b7cf-4f08-85ff-f274be9562b5","Type":"ContainerStarted","Data":"dd2ca1801b54deace4aa6aac1cd1f75d5cc2972de64e95425b10bb8b14719ace"} Nov 03 20:48:53.406766 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:48:53.406733 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c9n6g" event={"ID":"5e264222-b7cf-4f08-85ff-f274be9562b5","Type":"ContainerStarted","Data":"fee8881e67dc5b479710f8b432b7cd700ba92e56058d78211f549d0bc52ed638"} Nov 03 20:48:53.526287 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:48:53.526235 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c9n6g" podStartSLOduration=2.688187138 podStartE2EDuration="4.526219482s" totalImagesPullingTime="1.838032344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:48:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:48:53.523130984 +0000 UTC m=+568.748450798" watchObservedRunningTime="2026-11-03 20:48:53.526219482 +0000 UTC m=+568.751539272" Nov 03 20:48:54.212849 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:48:54.212810 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c9n6g" Nov 03 20:48:54.640084 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:48:54.640003 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 20:49:00.212897 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:00.212854 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c9n6g" Nov 03 20:49:00.225686 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:00.225659 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c9n6g" Nov 03 20:49:00.236395 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:00.236363 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw"] Nov 03 20:49:00.240052 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:00.240031 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" Nov 03 20:49:00.292404 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:00.292376 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Nov 03 20:49:00.298615 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:00.298583 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw"] Nov 03 20:49:00.359242 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:00.359208 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/006c5eae-06af-447b-a1d8-cd719cce2a92-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw\" (UID: \"006c5eae-06af-447b-a1d8-cd719cce2a92\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" Nov 03 20:49:00.359242 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:00.359244 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/006c5eae-06af-447b-a1d8-cd719cce2a92-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw\" (UID: \"006c5eae-06af-447b-a1d8-cd719cce2a92\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" Nov 03 20:49:00.359507 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:00.359267 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/006c5eae-06af-447b-a1d8-cd719cce2a92-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw\" (UID: \"006c5eae-06af-447b-a1d8-cd719cce2a92\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" Nov 03 20:49:00.359507 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:00.359480 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/006c5eae-06af-447b-a1d8-cd719cce2a92-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw\" (UID: \"006c5eae-06af-447b-a1d8-cd719cce2a92\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" Nov 03 20:49:00.359578 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:00.359517 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7h4f\" (UniqueName: \"kubernetes.io/projected/006c5eae-06af-447b-a1d8-cd719cce2a92-kube-api-access-q7h4f\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw\" (UID: \"006c5eae-06af-447b-a1d8-cd719cce2a92\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" Nov 03 20:49:00.359613 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:00.359592 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/006c5eae-06af-447b-a1d8-cd719cce2a92-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw\" (UID: \"006c5eae-06af-447b-a1d8-cd719cce2a92\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" Nov 03 20:49:00.448916 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:00.448883 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c9n6g" Nov 03 20:49:00.461057 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:00.461024 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/006c5eae-06af-447b-a1d8-cd719cce2a92-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw\" (UID: \"006c5eae-06af-447b-a1d8-cd719cce2a92\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" Nov 03 20:49:00.461259 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:00.461066 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7h4f\" (UniqueName: \"kubernetes.io/projected/006c5eae-06af-447b-a1d8-cd719cce2a92-kube-api-access-q7h4f\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw\" (UID: \"006c5eae-06af-447b-a1d8-cd719cce2a92\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" Nov 03 20:49:00.461259 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:00.461100 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/006c5eae-06af-447b-a1d8-cd719cce2a92-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw\" (UID: \"006c5eae-06af-447b-a1d8-cd719cce2a92\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" Nov 03 20:49:00.461259 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:00.461130 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/006c5eae-06af-447b-a1d8-cd719cce2a92-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw\" (UID: \"006c5eae-06af-447b-a1d8-cd719cce2a92\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" Nov 03 20:49:00.461259 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:00.461147 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/006c5eae-06af-447b-a1d8-cd719cce2a92-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw\" (UID: \"006c5eae-06af-447b-a1d8-cd719cce2a92\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" Nov 03 20:49:00.461259 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:00.461163 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/006c5eae-06af-447b-a1d8-cd719cce2a92-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw\" (UID: \"006c5eae-06af-447b-a1d8-cd719cce2a92\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" Nov 03 20:49:00.461598 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:00.461571 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/006c5eae-06af-447b-a1d8-cd719cce2a92-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw\" (UID: \"006c5eae-06af-447b-a1d8-cd719cce2a92\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" Nov 03 20:49:00.461663 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:00.461595 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/006c5eae-06af-447b-a1d8-cd719cce2a92-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw\" (UID: \"006c5eae-06af-447b-a1d8-cd719cce2a92\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" Nov 03 20:49:00.461663 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:00.461650 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/006c5eae-06af-447b-a1d8-cd719cce2a92-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw\" (UID: \"006c5eae-06af-447b-a1d8-cd719cce2a92\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" Nov 03 20:49:00.463369 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:00.463329 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/006c5eae-06af-447b-a1d8-cd719cce2a92-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw\" (UID: \"006c5eae-06af-447b-a1d8-cd719cce2a92\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" Nov 03 20:49:00.463649 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:00.463634 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/006c5eae-06af-447b-a1d8-cd719cce2a92-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw\" (UID: \"006c5eae-06af-447b-a1d8-cd719cce2a92\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" Nov 03 20:49:00.521882 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:00.521841 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7h4f\" (UniqueName: \"kubernetes.io/projected/006c5eae-06af-447b-a1d8-cd719cce2a92-kube-api-access-q7h4f\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw\" (UID: \"006c5eae-06af-447b-a1d8-cd719cce2a92\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" Nov 03 20:49:00.551824 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:00.551791 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" Nov 03 20:49:00.941013 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:00.940806 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw"] Nov 03 20:49:00.944006 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:49:00.943980 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod006c5eae_06af_447b_a1d8_cd719cce2a92.slice/crio-1f7ceab73baa6dfbb461dc4aeb0aa825ad61058c7c6eb04dd7e10dcee8fe1c71 WatchSource:0}: Error finding container 1f7ceab73baa6dfbb461dc4aeb0aa825ad61058c7c6eb04dd7e10dcee8fe1c71: Status 404 returned error can't find the container with id 1f7ceab73baa6dfbb461dc4aeb0aa825ad61058c7c6eb04dd7e10dcee8fe1c71 Nov 03 20:49:01.443937 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:01.443896 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" event={"ID":"006c5eae-06af-447b-a1d8-cd719cce2a92","Type":"ContainerStarted","Data":"1f7ceab73baa6dfbb461dc4aeb0aa825ad61058c7c6eb04dd7e10dcee8fe1c71"} Nov 03 20:49:02.450899 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:02.450867 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" event={"ID":"006c5eae-06af-447b-a1d8-cd719cce2a92","Type":"ContainerStarted","Data":"2f768d8de5b203a440c281db69e6e327a7d251cd284bb6c5e485efbfd170c0f8"} Nov 03 20:49:02.552767 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:02.552736 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" Nov 03 20:49:02.569489 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:02.569418 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" podStartSLOduration=1.950020584 podStartE2EDuration="2.569403739s" totalImagesPullingTime="619.383155ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:49:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:49:02.567286385 +0000 UTC m=+577.792606176" watchObservedRunningTime="2026-11-03 20:49:02.569403739 +0000 UTC m=+577.794723528" Nov 03 20:49:06.072161 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:06.072123 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 20:49:06.550582 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:06.550544 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-bblkx"] Nov 03 20:49:06.556877 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:06.556799 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bblkx" Nov 03 20:49:06.625352 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:06.625316 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6ab5742-35d8-4f39-8faf-2cf2c209e5a2-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bblkx\" (UID: \"d6ab5742-35d8-4f39-8faf-2cf2c209e5a2\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bblkx" Nov 03 20:49:06.625690 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:06.625666 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6ab5742-35d8-4f39-8faf-2cf2c209e5a2-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bblkx\" (UID: \"d6ab5742-35d8-4f39-8faf-2cf2c209e5a2\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bblkx" Nov 03 20:49:06.625900 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:06.625882 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6ab5742-35d8-4f39-8faf-2cf2c209e5a2-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bblkx\" (UID: \"d6ab5742-35d8-4f39-8faf-2cf2c209e5a2\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bblkx" Nov 03 20:49:06.626083 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:06.626067 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6ab5742-35d8-4f39-8faf-2cf2c209e5a2-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bblkx\" (UID: \"d6ab5742-35d8-4f39-8faf-2cf2c209e5a2\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bblkx" Nov 03 20:49:06.626251 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:06.626233 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx4ml\" (UniqueName: \"kubernetes.io/projected/d6ab5742-35d8-4f39-8faf-2cf2c209e5a2-kube-api-access-qx4ml\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bblkx\" (UID: \"d6ab5742-35d8-4f39-8faf-2cf2c209e5a2\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bblkx" Nov 03 20:49:06.626423 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:06.626405 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6ab5742-35d8-4f39-8faf-2cf2c209e5a2-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bblkx\" (UID: \"d6ab5742-35d8-4f39-8faf-2cf2c209e5a2\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bblkx" Nov 03 20:49:06.660042 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:06.660014 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Nov 03 20:49:06.662921 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:06.662893 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-bblkx"] Nov 03 20:49:06.727610 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:06.727569 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx4ml\" (UniqueName: \"kubernetes.io/projected/d6ab5742-35d8-4f39-8faf-2cf2c209e5a2-kube-api-access-qx4ml\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bblkx\" (UID: \"d6ab5742-35d8-4f39-8faf-2cf2c209e5a2\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bblkx" Nov 03 20:49:06.727610 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:06.727618 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6ab5742-35d8-4f39-8faf-2cf2c209e5a2-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bblkx\" (UID: \"d6ab5742-35d8-4f39-8faf-2cf2c209e5a2\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bblkx" Nov 03 20:49:06.727924 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:06.727688 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6ab5742-35d8-4f39-8faf-2cf2c209e5a2-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bblkx\" (UID: \"d6ab5742-35d8-4f39-8faf-2cf2c209e5a2\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bblkx" Nov 03 20:49:06.727924 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:06.727722 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6ab5742-35d8-4f39-8faf-2cf2c209e5a2-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bblkx\" (UID: \"d6ab5742-35d8-4f39-8faf-2cf2c209e5a2\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bblkx" Nov 03 20:49:06.727924 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:06.727760 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6ab5742-35d8-4f39-8faf-2cf2c209e5a2-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bblkx\" (UID: \"d6ab5742-35d8-4f39-8faf-2cf2c209e5a2\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bblkx" Nov 03 20:49:06.727924 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:06.727801 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6ab5742-35d8-4f39-8faf-2cf2c209e5a2-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bblkx\" (UID: \"d6ab5742-35d8-4f39-8faf-2cf2c209e5a2\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bblkx" Nov 03 20:49:06.728263 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:06.728238 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6ab5742-35d8-4f39-8faf-2cf2c209e5a2-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bblkx\" (UID: \"d6ab5742-35d8-4f39-8faf-2cf2c209e5a2\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bblkx" Nov 03 20:49:06.728334 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:06.728309 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6ab5742-35d8-4f39-8faf-2cf2c209e5a2-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bblkx\" (UID: \"d6ab5742-35d8-4f39-8faf-2cf2c209e5a2\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bblkx" Nov 03 20:49:06.728541 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:06.728512 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6ab5742-35d8-4f39-8faf-2cf2c209e5a2-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bblkx\" (UID: \"d6ab5742-35d8-4f39-8faf-2cf2c209e5a2\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bblkx" Nov 03 20:49:06.731089 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:06.731060 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6ab5742-35d8-4f39-8faf-2cf2c209e5a2-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bblkx\" (UID: \"d6ab5742-35d8-4f39-8faf-2cf2c209e5a2\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bblkx" Nov 03 20:49:06.731322 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:06.731300 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6ab5742-35d8-4f39-8faf-2cf2c209e5a2-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bblkx\" (UID: \"d6ab5742-35d8-4f39-8faf-2cf2c209e5a2\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bblkx" Nov 03 20:49:06.789456 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:06.789408 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx4ml\" (UniqueName: \"kubernetes.io/projected/d6ab5742-35d8-4f39-8faf-2cf2c209e5a2-kube-api-access-qx4ml\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-bblkx\" (UID: \"d6ab5742-35d8-4f39-8faf-2cf2c209e5a2\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bblkx" Nov 03 20:49:06.885753 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:06.885413 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bblkx" Nov 03 20:49:07.276561 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:07.276412 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-bblkx"] Nov 03 20:49:07.280375 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:49:07.280343 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6ab5742_35d8_4f39_8faf_2cf2c209e5a2.slice/crio-2ce4f7f3df35a8a3e32814759c1028bd1502604034ae8ca6c6432683d73a8904 WatchSource:0}: Error finding container 2ce4f7f3df35a8a3e32814759c1028bd1502604034ae8ca6c6432683d73a8904: Status 404 returned error can't find the container with id 2ce4f7f3df35a8a3e32814759c1028bd1502604034ae8ca6c6432683d73a8904 Nov 03 20:49:07.474657 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:07.474623 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bblkx" event={"ID":"d6ab5742-35d8-4f39-8faf-2cf2c209e5a2","Type":"ContainerStarted","Data":"2ce4f7f3df35a8a3e32814759c1028bd1502604034ae8ca6c6432683d73a8904"} Nov 03 20:49:08.481885 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:08.481849 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bblkx" event={"ID":"d6ab5742-35d8-4f39-8faf-2cf2c209e5a2","Type":"ContainerStarted","Data":"08eeba66dbd517574d02950963c24eb586456ae1164ae81070b1e06343d2b6a5"} Nov 03 20:49:08.600052 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:08.600004 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bblkx" podStartSLOduration=1.9570080330000001 podStartE2EDuration="2.599991368s" totalImagesPullingTime="642.983335ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:49:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:49:08.598647972 +0000 UTC m=+583.823967783" watchObservedRunningTime="2026-11-03 20:49:08.599991368 +0000 UTC m=+583.825311158" Nov 03 20:49:08.886465 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:08.886359 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bblkx" Nov 03 20:49:10.552188 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:10.552153 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" Nov 03 20:49:10.564544 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:10.564518 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" Nov 03 20:49:11.512059 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:11.512024 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" Nov 03 20:49:13.380496 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:13.380463 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-jjtc9"] Nov 03 20:49:13.384005 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:13.383982 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jjtc9" Nov 03 20:49:13.442319 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:13.441189 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Nov 03 20:49:13.492043 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:13.492003 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b0ee48b-a5e3-4989-8b7c-210a16e601be-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jjtc9\" (UID: \"8b0ee48b-a5e3-4989-8b7c-210a16e601be\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jjtc9" Nov 03 20:49:13.492222 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:13.492058 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b0ee48b-a5e3-4989-8b7c-210a16e601be-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jjtc9\" (UID: \"8b0ee48b-a5e3-4989-8b7c-210a16e601be\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jjtc9" Nov 03 20:49:13.492222 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:13.492173 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qmsq\" (UniqueName: \"kubernetes.io/projected/8b0ee48b-a5e3-4989-8b7c-210a16e601be-kube-api-access-5qmsq\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jjtc9\" (UID: \"8b0ee48b-a5e3-4989-8b7c-210a16e601be\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jjtc9" Nov 03 20:49:13.492307 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:13.492281 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b0ee48b-a5e3-4989-8b7c-210a16e601be-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jjtc9\" (UID: \"8b0ee48b-a5e3-4989-8b7c-210a16e601be\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jjtc9" Nov 03 20:49:13.492377 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:13.492360 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b0ee48b-a5e3-4989-8b7c-210a16e601be-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jjtc9\" (UID: \"8b0ee48b-a5e3-4989-8b7c-210a16e601be\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jjtc9" Nov 03 20:49:13.492454 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:13.492422 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b0ee48b-a5e3-4989-8b7c-210a16e601be-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jjtc9\" (UID: \"8b0ee48b-a5e3-4989-8b7c-210a16e601be\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jjtc9" Nov 03 20:49:13.493497 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:13.493475 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-jjtc9"] Nov 03 20:49:13.593399 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:13.593361 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b0ee48b-a5e3-4989-8b7c-210a16e601be-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jjtc9\" (UID: \"8b0ee48b-a5e3-4989-8b7c-210a16e601be\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jjtc9" Nov 03 20:49:13.593600 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:13.593489 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b0ee48b-a5e3-4989-8b7c-210a16e601be-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jjtc9\" (UID: \"8b0ee48b-a5e3-4989-8b7c-210a16e601be\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jjtc9" Nov 03 20:49:13.593600 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:13.593526 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b0ee48b-a5e3-4989-8b7c-210a16e601be-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jjtc9\" (UID: \"8b0ee48b-a5e3-4989-8b7c-210a16e601be\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jjtc9" Nov 03 20:49:13.593600 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:13.593552 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qmsq\" (UniqueName: \"kubernetes.io/projected/8b0ee48b-a5e3-4989-8b7c-210a16e601be-kube-api-access-5qmsq\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jjtc9\" (UID: \"8b0ee48b-a5e3-4989-8b7c-210a16e601be\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jjtc9" Nov 03 20:49:13.593771 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:13.593610 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b0ee48b-a5e3-4989-8b7c-210a16e601be-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jjtc9\" (UID: \"8b0ee48b-a5e3-4989-8b7c-210a16e601be\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jjtc9" Nov 03 20:49:13.593771 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:13.593673 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b0ee48b-a5e3-4989-8b7c-210a16e601be-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jjtc9\" (UID: \"8b0ee48b-a5e3-4989-8b7c-210a16e601be\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jjtc9" Nov 03 20:49:13.593991 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:13.593966 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b0ee48b-a5e3-4989-8b7c-210a16e601be-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jjtc9\" (UID: \"8b0ee48b-a5e3-4989-8b7c-210a16e601be\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jjtc9" Nov 03 20:49:13.594056 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:13.593991 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b0ee48b-a5e3-4989-8b7c-210a16e601be-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jjtc9\" (UID: \"8b0ee48b-a5e3-4989-8b7c-210a16e601be\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jjtc9" Nov 03 20:49:13.594056 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:13.594035 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b0ee48b-a5e3-4989-8b7c-210a16e601be-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jjtc9\" (UID: \"8b0ee48b-a5e3-4989-8b7c-210a16e601be\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jjtc9" Nov 03 20:49:13.595939 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:13.595914 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b0ee48b-a5e3-4989-8b7c-210a16e601be-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jjtc9\" (UID: \"8b0ee48b-a5e3-4989-8b7c-210a16e601be\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jjtc9" Nov 03 20:49:13.596099 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:13.596079 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b0ee48b-a5e3-4989-8b7c-210a16e601be-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jjtc9\" (UID: \"8b0ee48b-a5e3-4989-8b7c-210a16e601be\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jjtc9" Nov 03 20:49:13.654603 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:13.654569 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qmsq\" (UniqueName: \"kubernetes.io/projected/8b0ee48b-a5e3-4989-8b7c-210a16e601be-kube-api-access-5qmsq\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-jjtc9\" (UID: \"8b0ee48b-a5e3-4989-8b7c-210a16e601be\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jjtc9" Nov 03 20:49:13.696512 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:13.696463 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jjtc9" Nov 03 20:49:14.071491 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:14.071465 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-jjtc9"] Nov 03 20:49:14.074046 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:49:14.074018 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b0ee48b_a5e3_4989_8b7c_210a16e601be.slice/crio-8b741abd8efaeac35d287d2bfe6ff60c986997c1a49927c3e33033a6ecc677d7 WatchSource:0}: Error finding container 8b741abd8efaeac35d287d2bfe6ff60c986997c1a49927c3e33033a6ecc677d7: Status 404 returned error can't find the container with id 8b741abd8efaeac35d287d2bfe6ff60c986997c1a49927c3e33033a6ecc677d7 Nov 03 20:49:14.514557 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:14.514513 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jjtc9" event={"ID":"8b0ee48b-a5e3-4989-8b7c-210a16e601be","Type":"ContainerStarted","Data":"8b741abd8efaeac35d287d2bfe6ff60c986997c1a49927c3e33033a6ecc677d7"} Nov 03 20:49:15.520640 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:15.520606 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jjtc9" event={"ID":"8b0ee48b-a5e3-4989-8b7c-210a16e601be","Type":"ContainerStarted","Data":"9d958cfe995a5a8f0c55bf6c2348ac42acfa6a67c653836819843782ad0ca9d4"} Nov 03 20:49:15.638623 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:15.638564 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jjtc9" podStartSLOduration=1.985654985 podStartE2EDuration="2.638548657s" totalImagesPullingTime="652.893672ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:49:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:49:15.637517069 +0000 UTC m=+590.862836874" watchObservedRunningTime="2026-11-03 20:49:15.638548657 +0000 UTC m=+590.863868446" Nov 03 20:49:15.697589 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:15.697544 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jjtc9" Nov 03 20:49:16.320860 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:49:16.320826 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 20:49:16.886717 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:16.886685 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bblkx" Nov 03 20:49:16.899501 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:16.899480 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bblkx" Nov 03 20:49:17.079778 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:17.079740 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pblg"] Nov 03 20:49:17.083398 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:17.083376 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pblg" Nov 03 20:49:17.127241 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:17.127207 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl877\" (UniqueName: \"kubernetes.io/projected/997eb6ae-52a1-4600-b955-59bdea0a6da3-kube-api-access-zl877\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pblg\" (UID: \"997eb6ae-52a1-4600-b955-59bdea0a6da3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pblg" Nov 03 20:49:17.127461 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:17.127265 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/997eb6ae-52a1-4600-b955-59bdea0a6da3-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pblg\" (UID: \"997eb6ae-52a1-4600-b955-59bdea0a6da3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pblg" Nov 03 20:49:17.127461 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:17.127423 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/997eb6ae-52a1-4600-b955-59bdea0a6da3-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pblg\" (UID: \"997eb6ae-52a1-4600-b955-59bdea0a6da3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pblg" Nov 03 20:49:17.127593 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:17.127528 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/997eb6ae-52a1-4600-b955-59bdea0a6da3-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pblg\" (UID: \"997eb6ae-52a1-4600-b955-59bdea0a6da3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pblg" Nov 03 20:49:17.127593 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:17.127570 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/997eb6ae-52a1-4600-b955-59bdea0a6da3-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pblg\" (UID: \"997eb6ae-52a1-4600-b955-59bdea0a6da3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pblg" Nov 03 20:49:17.127700 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:17.127606 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/997eb6ae-52a1-4600-b955-59bdea0a6da3-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pblg\" (UID: \"997eb6ae-52a1-4600-b955-59bdea0a6da3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pblg" Nov 03 20:49:17.136408 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:17.136383 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Nov 03 20:49:17.142250 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:17.142228 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pblg"] Nov 03 20:49:17.228834 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:17.228788 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/997eb6ae-52a1-4600-b955-59bdea0a6da3-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pblg\" (UID: \"997eb6ae-52a1-4600-b955-59bdea0a6da3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pblg" Nov 03 20:49:17.229062 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:17.228850 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/997eb6ae-52a1-4600-b955-59bdea0a6da3-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pblg\" (UID: \"997eb6ae-52a1-4600-b955-59bdea0a6da3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pblg" Nov 03 20:49:17.229062 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:17.228887 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/997eb6ae-52a1-4600-b955-59bdea0a6da3-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pblg\" (UID: \"997eb6ae-52a1-4600-b955-59bdea0a6da3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pblg" Nov 03 20:49:17.229062 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:17.228918 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/997eb6ae-52a1-4600-b955-59bdea0a6da3-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pblg\" (UID: \"997eb6ae-52a1-4600-b955-59bdea0a6da3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pblg" Nov 03 20:49:17.229062 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:17.228942 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/997eb6ae-52a1-4600-b955-59bdea0a6da3-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pblg\" (UID: \"997eb6ae-52a1-4600-b955-59bdea0a6da3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pblg" Nov 03 20:49:17.229062 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:17.228996 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl877\" (UniqueName: \"kubernetes.io/projected/997eb6ae-52a1-4600-b955-59bdea0a6da3-kube-api-access-zl877\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pblg\" (UID: \"997eb6ae-52a1-4600-b955-59bdea0a6da3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pblg" Nov 03 20:49:17.232307 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:17.231369 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/997eb6ae-52a1-4600-b955-59bdea0a6da3-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pblg\" (UID: \"997eb6ae-52a1-4600-b955-59bdea0a6da3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pblg" Nov 03 20:49:17.232307 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:17.231371 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/997eb6ae-52a1-4600-b955-59bdea0a6da3-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pblg\" (UID: \"997eb6ae-52a1-4600-b955-59bdea0a6da3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pblg" Nov 03 20:49:17.232307 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:17.231657 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/997eb6ae-52a1-4600-b955-59bdea0a6da3-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pblg\" (UID: \"997eb6ae-52a1-4600-b955-59bdea0a6da3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pblg" Nov 03 20:49:17.234138 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:17.234113 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/997eb6ae-52a1-4600-b955-59bdea0a6da3-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pblg\" (UID: \"997eb6ae-52a1-4600-b955-59bdea0a6da3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pblg" Nov 03 20:49:17.234786 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:17.234765 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/997eb6ae-52a1-4600-b955-59bdea0a6da3-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pblg\" (UID: \"997eb6ae-52a1-4600-b955-59bdea0a6da3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pblg" Nov 03 20:49:17.292172 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:17.292135 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl877\" (UniqueName: \"kubernetes.io/projected/997eb6ae-52a1-4600-b955-59bdea0a6da3-kube-api-access-zl877\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pblg\" (UID: \"997eb6ae-52a1-4600-b955-59bdea0a6da3\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pblg" Nov 03 20:49:17.395952 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:17.395846 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pblg" Nov 03 20:49:17.547667 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:17.547632 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-bblkx" Nov 03 20:49:17.781113 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:17.781091 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pblg"] Nov 03 20:49:17.783208 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:49:17.783184 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod997eb6ae_52a1_4600_b955_59bdea0a6da3.slice/crio-618ed0298f8b3eea0d8c6c09ffabfa4dc1ebf3a1df4e5154c08ea349bb20b0ba WatchSource:0}: Error finding container 618ed0298f8b3eea0d8c6c09ffabfa4dc1ebf3a1df4e5154c08ea349bb20b0ba: Status 404 returned error can't find the container with id 618ed0298f8b3eea0d8c6c09ffabfa4dc1ebf3a1df4e5154c08ea349bb20b0ba Nov 03 20:49:18.537862 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:18.537824 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pblg" event={"ID":"997eb6ae-52a1-4600-b955-59bdea0a6da3","Type":"ContainerStarted","Data":"2230a2e987d9d5fe4950f5b5e4dcc04c0c21cbb49ac181c3b5aa1136f817e854"} Nov 03 20:49:18.538427 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:18.537879 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pblg" event={"ID":"997eb6ae-52a1-4600-b955-59bdea0a6da3","Type":"ContainerStarted","Data":"618ed0298f8b3eea0d8c6c09ffabfa4dc1ebf3a1df4e5154c08ea349bb20b0ba"} Nov 03 20:49:18.656177 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:18.656126 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pblg" podStartSLOduration=1.049889498 podStartE2EDuration="1.65611194s" totalImagesPullingTime="606.222442ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:49:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:49:18.654401263 +0000 UTC m=+593.879721054" watchObservedRunningTime="2026-11-03 20:49:18.65611194 +0000 UTC m=+593.881431731" Nov 03 20:49:19.396717 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:19.396675 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pblg" Nov 03 20:49:20.131309 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:20.131268 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 20:49:23.697640 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:23.697593 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jjtc9" Nov 03 20:49:23.709829 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:23.709796 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jjtc9" Nov 03 20:49:24.585342 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:24.585316 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-jjtc9" Nov 03 20:49:27.396754 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:27.396719 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pblg" Nov 03 20:49:27.409338 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:27.409309 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pblg" Nov 03 20:49:27.600502 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:27.600476 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pblg" Nov 03 20:49:31.372322 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:31.372285 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jccsf"] Nov 03 20:49:31.377710 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:31.377694 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jccsf" Nov 03 20:49:31.477093 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:31.477060 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc5b2941-a16d-4ab5-a36b-12a7f381c8c0-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jccsf\" (UID: \"bc5b2941-a16d-4ab5-a36b-12a7f381c8c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jccsf" Nov 03 20:49:31.477285 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:31.477129 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc5b2941-a16d-4ab5-a36b-12a7f381c8c0-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jccsf\" (UID: \"bc5b2941-a16d-4ab5-a36b-12a7f381c8c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jccsf" Nov 03 20:49:31.477285 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:31.477187 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc5b2941-a16d-4ab5-a36b-12a7f381c8c0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jccsf\" (UID: \"bc5b2941-a16d-4ab5-a36b-12a7f381c8c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jccsf" Nov 03 20:49:31.477285 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:31.477261 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc5b2941-a16d-4ab5-a36b-12a7f381c8c0-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jccsf\" (UID: \"bc5b2941-a16d-4ab5-a36b-12a7f381c8c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jccsf" Nov 03 20:49:31.477504 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:31.477322 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng4f6\" (UniqueName: \"kubernetes.io/projected/bc5b2941-a16d-4ab5-a36b-12a7f381c8c0-kube-api-access-ng4f6\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jccsf\" (UID: \"bc5b2941-a16d-4ab5-a36b-12a7f381c8c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jccsf" Nov 03 20:49:31.477504 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:31.477342 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc5b2941-a16d-4ab5-a36b-12a7f381c8c0-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jccsf\" (UID: \"bc5b2941-a16d-4ab5-a36b-12a7f381c8c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jccsf" Nov 03 20:49:31.477504 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:31.477359 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc5b2941-a16d-4ab5-a36b-12a7f381c8c0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jccsf\" (UID: \"bc5b2941-a16d-4ab5-a36b-12a7f381c8c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jccsf" Nov 03 20:49:31.477504 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:31.477427 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc5b2941-a16d-4ab5-a36b-12a7f381c8c0-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jccsf\" (UID: \"bc5b2941-a16d-4ab5-a36b-12a7f381c8c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jccsf" Nov 03 20:49:31.477719 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:31.477507 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc5b2941-a16d-4ab5-a36b-12a7f381c8c0-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jccsf\" (UID: \"bc5b2941-a16d-4ab5-a36b-12a7f381c8c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jccsf" Nov 03 20:49:31.578587 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:31.578558 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc5b2941-a16d-4ab5-a36b-12a7f381c8c0-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jccsf\" (UID: \"bc5b2941-a16d-4ab5-a36b-12a7f381c8c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jccsf" Nov 03 20:49:31.578778 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:31.578612 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc5b2941-a16d-4ab5-a36b-12a7f381c8c0-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jccsf\" (UID: \"bc5b2941-a16d-4ab5-a36b-12a7f381c8c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jccsf" Nov 03 20:49:31.578778 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:31.578642 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc5b2941-a16d-4ab5-a36b-12a7f381c8c0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jccsf\" (UID: \"bc5b2941-a16d-4ab5-a36b-12a7f381c8c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jccsf" Nov 03 20:49:31.578778 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:31.578668 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc5b2941-a16d-4ab5-a36b-12a7f381c8c0-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jccsf\" (UID: \"bc5b2941-a16d-4ab5-a36b-12a7f381c8c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jccsf" Nov 03 20:49:31.578778 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:31.578700 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng4f6\" (UniqueName: \"kubernetes.io/projected/bc5b2941-a16d-4ab5-a36b-12a7f381c8c0-kube-api-access-ng4f6\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jccsf\" (UID: \"bc5b2941-a16d-4ab5-a36b-12a7f381c8c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jccsf" Nov 03 20:49:31.578778 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:31.578727 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc5b2941-a16d-4ab5-a36b-12a7f381c8c0-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jccsf\" (UID: \"bc5b2941-a16d-4ab5-a36b-12a7f381c8c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jccsf" Nov 03 20:49:31.578778 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:31.578750 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc5b2941-a16d-4ab5-a36b-12a7f381c8c0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jccsf\" (UID: \"bc5b2941-a16d-4ab5-a36b-12a7f381c8c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jccsf" Nov 03 20:49:31.579082 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:31.578793 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc5b2941-a16d-4ab5-a36b-12a7f381c8c0-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jccsf\" (UID: \"bc5b2941-a16d-4ab5-a36b-12a7f381c8c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jccsf" Nov 03 20:49:31.579082 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:31.578842 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc5b2941-a16d-4ab5-a36b-12a7f381c8c0-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jccsf\" (UID: \"bc5b2941-a16d-4ab5-a36b-12a7f381c8c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jccsf" Nov 03 20:49:31.579198 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:31.579182 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc5b2941-a16d-4ab5-a36b-12a7f381c8c0-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jccsf\" (UID: \"bc5b2941-a16d-4ab5-a36b-12a7f381c8c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jccsf" Nov 03 20:49:31.579248 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:31.579215 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc5b2941-a16d-4ab5-a36b-12a7f381c8c0-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jccsf\" (UID: \"bc5b2941-a16d-4ab5-a36b-12a7f381c8c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jccsf" Nov 03 20:49:31.579390 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:31.579365 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc5b2941-a16d-4ab5-a36b-12a7f381c8c0-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jccsf\" (UID: \"bc5b2941-a16d-4ab5-a36b-12a7f381c8c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jccsf" Nov 03 20:49:31.579640 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:31.579446 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc5b2941-a16d-4ab5-a36b-12a7f381c8c0-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jccsf\" (UID: \"bc5b2941-a16d-4ab5-a36b-12a7f381c8c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jccsf" Nov 03 20:49:31.579754 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:31.579734 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc5b2941-a16d-4ab5-a36b-12a7f381c8c0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jccsf\" (UID: \"bc5b2941-a16d-4ab5-a36b-12a7f381c8c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jccsf" Nov 03 20:49:31.581195 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:31.581169 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc5b2941-a16d-4ab5-a36b-12a7f381c8c0-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jccsf\" (UID: \"bc5b2941-a16d-4ab5-a36b-12a7f381c8c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jccsf" Nov 03 20:49:31.581384 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:31.581367 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc5b2941-a16d-4ab5-a36b-12a7f381c8c0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jccsf\" (UID: \"bc5b2941-a16d-4ab5-a36b-12a7f381c8c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jccsf" Nov 03 20:49:31.588008 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:31.587977 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jccsf"] Nov 03 20:49:31.639416 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:31.639345 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc5b2941-a16d-4ab5-a36b-12a7f381c8c0-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jccsf\" (UID: \"bc5b2941-a16d-4ab5-a36b-12a7f381c8c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jccsf" Nov 03 20:49:31.639594 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:31.639546 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng4f6\" (UniqueName: \"kubernetes.io/projected/bc5b2941-a16d-4ab5-a36b-12a7f381c8c0-kube-api-access-ng4f6\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-jccsf\" (UID: \"bc5b2941-a16d-4ab5-a36b-12a7f381c8c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jccsf" Nov 03 20:49:31.691102 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:31.691073 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jccsf" Nov 03 20:49:32.188351 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:32.188317 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jccsf"] Nov 03 20:49:32.214673 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:49:32.214647 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc5b2941_a16d_4ab5_a36b_12a7f381c8c0.slice/crio-b89fba9c5e754f8f718b5f3991377fe1322522ff87753e4de7189f116f3fd15c WatchSource:0}: Error finding container b89fba9c5e754f8f718b5f3991377fe1322522ff87753e4de7189f116f3fd15c: Status 404 returned error can't find the container with id b89fba9c5e754f8f718b5f3991377fe1322522ff87753e4de7189f116f3fd15c Nov 03 20:49:32.222863 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:32.222826 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:49:32.222977 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:32.222891 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:49:32.222977 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:32.222918 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:49:32.615896 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:32.615780 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jccsf" event={"ID":"bc5b2941-a16d-4ab5-a36b-12a7f381c8c0","Type":"ContainerStarted","Data":"a7c6681dd4637d18897540b6f35b4bc5d27fdab63018f303004341edd08f427b"} Nov 03 20:49:32.615896 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:32.615869 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jccsf" event={"ID":"bc5b2941-a16d-4ab5-a36b-12a7f381c8c0","Type":"ContainerStarted","Data":"b89fba9c5e754f8f718b5f3991377fe1322522ff87753e4de7189f116f3fd15c"} Nov 03 20:49:32.691383 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:32.691338 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jccsf" Nov 03 20:49:32.743269 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:32.743204 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jccsf" podStartSLOduration=1.74319042 podStartE2EDuration="1.74319042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:49:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:49:32.741610137 +0000 UTC m=+607.966929926" watchObservedRunningTime="2026-11-03 20:49:32.74319042 +0000 UTC m=+607.968510210" Nov 03 20:49:33.692626 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:33.692595 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-jccsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 03 20:49:33.693102 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:33.692671 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jccsf" podUID="bc5b2941-a16d-4ab5-a36b-12a7f381c8c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 03 20:49:34.692546 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:34.692508 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-jccsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 03 20:49:34.692814 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:34.692563 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jccsf" podUID="bc5b2941-a16d-4ab5-a36b-12a7f381c8c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 03 20:49:35.692240 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:35.692203 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-jccsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 03 20:49:35.692458 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:35.692258 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jccsf" podUID="bc5b2941-a16d-4ab5-a36b-12a7f381c8c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 03 20:49:36.691572 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:36.691533 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-jccsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 03 20:49:36.692000 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:36.691591 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jccsf" podUID="bc5b2941-a16d-4ab5-a36b-12a7f381c8c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 03 20:49:37.692349 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:37.692311 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-jccsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 03 20:49:37.692776 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:37.692367 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jccsf" podUID="bc5b2941-a16d-4ab5-a36b-12a7f381c8c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 03 20:49:38.692003 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:38.691972 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-jccsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 03 20:49:38.692191 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:38.692024 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jccsf" podUID="bc5b2941-a16d-4ab5-a36b-12a7f381c8c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 03 20:49:39.692004 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:39.691970 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-jccsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 03 20:49:39.692491 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:39.692028 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jccsf" podUID="bc5b2941-a16d-4ab5-a36b-12a7f381c8c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 03 20:49:40.695891 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:40.695844 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jccsf" Nov 03 20:49:40.696352 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:40.696137 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jccsf" Nov 03 20:49:40.696836 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:40.696819 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jccsf" Nov 03 20:49:46.320278 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:49:46.320235 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 20:49:48.338634 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:48.338601 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-857bc5d6df-rncbs"] Nov 03 20:49:48.339006 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:48.338792 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-857bc5d6df-rncbs" podUID="a060940f-3a28-4af9-9fa0-724b4ace3a2e" containerName="authorino" containerID="cri-o://777c0ee4dfb33bfe38d61acd5c0c2e89263436014db2f895762a6153898a0311" gracePeriod=30 Nov 03 20:49:48.592192 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:48.592124 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-857bc5d6df-rncbs" Nov 03 20:49:48.691064 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:48.691017 4305 generic.go:356] "Generic (PLEG): container finished" podID="a060940f-3a28-4af9-9fa0-724b4ace3a2e" containerID="777c0ee4dfb33bfe38d61acd5c0c2e89263436014db2f895762a6153898a0311" exitCode=0 Nov 03 20:49:48.691064 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:48.691070 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-857bc5d6df-rncbs" Nov 03 20:49:48.691295 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:48.691108 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-857bc5d6df-rncbs" event={"ID":"a060940f-3a28-4af9-9fa0-724b4ace3a2e","Type":"ContainerDied","Data":"777c0ee4dfb33bfe38d61acd5c0c2e89263436014db2f895762a6153898a0311"} Nov 03 20:49:48.691295 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:48.691152 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-857bc5d6df-rncbs" event={"ID":"a060940f-3a28-4af9-9fa0-724b4ace3a2e","Type":"ContainerDied","Data":"07b8e875ded65ac99cc2b534e51e9e2f40272f154c254e14edd3c14ab300b3f0"} Nov 03 20:49:48.691295 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:48.691169 4305 scope.go:122] "RemoveContainer" containerID="777c0ee4dfb33bfe38d61acd5c0c2e89263436014db2f895762a6153898a0311" Nov 03 20:49:48.699551 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:48.699533 4305 scope.go:122] "RemoveContainer" containerID="777c0ee4dfb33bfe38d61acd5c0c2e89263436014db2f895762a6153898a0311" Nov 03 20:49:48.699811 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:49:48.699794 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"777c0ee4dfb33bfe38d61acd5c0c2e89263436014db2f895762a6153898a0311\": container with ID starting with 777c0ee4dfb33bfe38d61acd5c0c2e89263436014db2f895762a6153898a0311 not found: ID does not exist" containerID="777c0ee4dfb33bfe38d61acd5c0c2e89263436014db2f895762a6153898a0311" Nov 03 20:49:48.699867 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:48.699822 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"777c0ee4dfb33bfe38d61acd5c0c2e89263436014db2f895762a6153898a0311"} err="failed to get container status \"777c0ee4dfb33bfe38d61acd5c0c2e89263436014db2f895762a6153898a0311\": rpc error: code = NotFound desc = could not find container \"777c0ee4dfb33bfe38d61acd5c0c2e89263436014db2f895762a6153898a0311\": container with ID starting with 777c0ee4dfb33bfe38d61acd5c0c2e89263436014db2f895762a6153898a0311 not found: ID does not exist" Nov 03 20:49:48.754333 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:48.754293 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a060940f-3a28-4af9-9fa0-724b4ace3a2e-kube-api-access-nk5jp\" (UniqueName: \"kubernetes.io/projected/a060940f-3a28-4af9-9fa0-724b4ace3a2e-kube-api-access-nk5jp\") pod \"a060940f-3a28-4af9-9fa0-724b4ace3a2e\" (UID: \"a060940f-3a28-4af9-9fa0-724b4ace3a2e\") " Nov 03 20:49:48.754565 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:48.754367 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a060940f-3a28-4af9-9fa0-724b4ace3a2e-tls-cert\" (UniqueName: \"kubernetes.io/secret/a060940f-3a28-4af9-9fa0-724b4ace3a2e-tls-cert\") pod \"a060940f-3a28-4af9-9fa0-724b4ace3a2e\" (UID: \"a060940f-3a28-4af9-9fa0-724b4ace3a2e\") " Nov 03 20:49:48.756698 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:48.756671 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a060940f-3a28-4af9-9fa0-724b4ace3a2e-kube-api-access-nk5jp" pod "a060940f-3a28-4af9-9fa0-724b4ace3a2e" (UID: "a060940f-3a28-4af9-9fa0-724b4ace3a2e"). InnerVolumeSpecName "kube-api-access-nk5jp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:49:48.765500 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:48.765474 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a060940f-3a28-4af9-9fa0-724b4ace3a2e-tls-cert" pod "a060940f-3a28-4af9-9fa0-724b4ace3a2e" (UID: "a060940f-3a28-4af9-9fa0-724b4ace3a2e"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:49:48.855558 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:48.855470 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nk5jp\" (UniqueName: \"kubernetes.io/projected/a060940f-3a28-4af9-9fa0-724b4ace3a2e-kube-api-access-nk5jp\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:49:48.855558 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:48.855505 4305 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a060940f-3a28-4af9-9fa0-724b4ace3a2e-tls-cert\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:49:49.104307 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:49.104270 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-857bc5d6df-rncbs"] Nov 03 20:49:49.153697 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:49.153665 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-857bc5d6df-rncbs"] Nov 03 20:49:50.894052 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:50.894019 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a060940f-3a28-4af9-9fa0-724b4ace3a2e" path="/var/lib/kubelet/pods/a060940f-3a28-4af9-9fa0-724b4ace3a2e/volumes" Nov 03 20:49:57.250205 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:57.250167 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-dctzh"] Nov 03 20:49:57.250603 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:57.250573 4305 cpu_manager.go:414] "containerMap: removing container" podUID="a060940f-3a28-4af9-9fa0-724b4ace3a2e" containerName="authorino" Nov 03 20:49:57.250603 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:57.250586 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a060940f-3a28-4af9-9fa0-724b4ace3a2e" containerName="authorino" Nov 03 20:49:57.250673 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:57.250657 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="a060940f-3a28-4af9-9fa0-724b4ace3a2e" containerName="authorino" Nov 03 20:49:57.254741 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:57.254724 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-dctzh" Nov 03 20:49:57.313343 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:57.313314 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 03 20:49:57.320911 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:57.320879 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-dctzh"] Nov 03 20:49:57.339073 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:57.339025 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4d05666-5bea-415a-b266-805722b1158f-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-dctzh\" (UID: \"d4d05666-5bea-415a-b266-805722b1158f\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-dctzh" Nov 03 20:49:57.339267 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:57.339097 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz486\" (UniqueName: \"kubernetes.io/projected/d4d05666-5bea-415a-b266-805722b1158f-kube-api-access-xz486\") pod \"payload-pre-processing-7b98bb678c-dctzh\" (UID: \"d4d05666-5bea-415a-b266-805722b1158f\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-dctzh" Nov 03 20:49:57.358746 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:57.358709 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-g4wrr\"" Nov 03 20:49:57.391852 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:57.391810 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-z6z42"] Nov 03 20:49:57.396009 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:57.395989 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7bb9489676-z6z42" Nov 03 20:49:57.440120 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:57.440086 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6725e63-2f8d-4f1b-88ad-41057dd4a5c2-plugins-config-volume\") pod \"payload-processing-7bb9489676-z6z42\" (UID: \"b6725e63-2f8d-4f1b-88ad-41057dd4a5c2\") " pod="openshift-ingress/payload-processing-7bb9489676-z6z42" Nov 03 20:49:57.440311 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:57.440139 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4d05666-5bea-415a-b266-805722b1158f-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-dctzh\" (UID: \"d4d05666-5bea-415a-b266-805722b1158f\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-dctzh" Nov 03 20:49:57.440311 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:57.440240 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz486\" (UniqueName: \"kubernetes.io/projected/d4d05666-5bea-415a-b266-805722b1158f-kube-api-access-xz486\") pod \"payload-pre-processing-7b98bb678c-dctzh\" (UID: \"d4d05666-5bea-415a-b266-805722b1158f\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-dctzh" Nov 03 20:49:57.440407 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:57.440320 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rkn4\" (UniqueName: \"kubernetes.io/projected/b6725e63-2f8d-4f1b-88ad-41057dd4a5c2-kube-api-access-8rkn4\") pod \"payload-processing-7bb9489676-z6z42\" (UID: \"b6725e63-2f8d-4f1b-88ad-41057dd4a5c2\") " pod="openshift-ingress/payload-processing-7bb9489676-z6z42" Nov 03 20:49:57.440925 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:57.440905 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4d05666-5bea-415a-b266-805722b1158f-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-dctzh\" (UID: \"d4d05666-5bea-415a-b266-805722b1158f\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-dctzh" Nov 03 20:49:57.453182 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:57.453087 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-z6z42"] Nov 03 20:49:57.501492 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:57.501378 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz486\" (UniqueName: \"kubernetes.io/projected/d4d05666-5bea-415a-b266-805722b1158f-kube-api-access-xz486\") pod \"payload-pre-processing-7b98bb678c-dctzh\" (UID: \"d4d05666-5bea-415a-b266-805722b1158f\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-dctzh" Nov 03 20:49:57.541933 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:57.541900 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6725e63-2f8d-4f1b-88ad-41057dd4a5c2-plugins-config-volume\") pod \"payload-processing-7bb9489676-z6z42\" (UID: \"b6725e63-2f8d-4f1b-88ad-41057dd4a5c2\") " pod="openshift-ingress/payload-processing-7bb9489676-z6z42" Nov 03 20:49:57.542143 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:57.541975 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rkn4\" (UniqueName: \"kubernetes.io/projected/b6725e63-2f8d-4f1b-88ad-41057dd4a5c2-kube-api-access-8rkn4\") pod \"payload-processing-7bb9489676-z6z42\" (UID: \"b6725e63-2f8d-4f1b-88ad-41057dd4a5c2\") " pod="openshift-ingress/payload-processing-7bb9489676-z6z42" Nov 03 20:49:57.542683 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:57.542658 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6725e63-2f8d-4f1b-88ad-41057dd4a5c2-plugins-config-volume\") pod \"payload-processing-7bb9489676-z6z42\" (UID: \"b6725e63-2f8d-4f1b-88ad-41057dd4a5c2\") " pod="openshift-ingress/payload-processing-7bb9489676-z6z42" Nov 03 20:49:57.567832 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:57.567803 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-dctzh" Nov 03 20:49:57.603571 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:57.603540 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rkn4\" (UniqueName: \"kubernetes.io/projected/b6725e63-2f8d-4f1b-88ad-41057dd4a5c2-kube-api-access-8rkn4\") pod \"payload-processing-7bb9489676-z6z42\" (UID: \"b6725e63-2f8d-4f1b-88ad-41057dd4a5c2\") " pod="openshift-ingress/payload-processing-7bb9489676-z6z42" Nov 03 20:49:57.712400 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:57.712367 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7bb9489676-z6z42" Nov 03 20:49:57.949545 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:57.949521 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-dctzh"] Nov 03 20:49:57.951183 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:49:57.951158 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4d05666_5bea_415a_b266_805722b1158f.slice/crio-3d97d7f7cfac2ba984d946ad27360d9fae8d0bcc4cf5c946720c6de021eb4424 WatchSource:0}: Error finding container 3d97d7f7cfac2ba984d946ad27360d9fae8d0bcc4cf5c946720c6de021eb4424: Status 404 returned error can't find the container with id 3d97d7f7cfac2ba984d946ad27360d9fae8d0bcc4cf5c946720c6de021eb4424 Nov 03 20:49:57.952936 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:57.952922 4305 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:49:58.091318 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:58.091294 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-z6z42"] Nov 03 20:49:58.093743 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:49:58.093718 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6725e63_2f8d_4f1b_88ad_41057dd4a5c2.slice/crio-0207512017d669270719b542e9d8096d3cf11947733c23d8112a2445027cb296 WatchSource:0}: Error finding container 0207512017d669270719b542e9d8096d3cf11947733c23d8112a2445027cb296: Status 404 returned error can't find the container with id 0207512017d669270719b542e9d8096d3cf11947733c23d8112a2445027cb296 Nov 03 20:49:58.737984 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:58.737955 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7bb9489676-z6z42" event={"ID":"b6725e63-2f8d-4f1b-88ad-41057dd4a5c2","Type":"ContainerStarted","Data":"0207512017d669270719b542e9d8096d3cf11947733c23d8112a2445027cb296"} Nov 03 20:49:58.739079 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:49:58.739046 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-dctzh" event={"ID":"d4d05666-5bea-415a-b266-805722b1158f","Type":"ContainerStarted","Data":"3d97d7f7cfac2ba984d946ad27360d9fae8d0bcc4cf5c946720c6de021eb4424"} Nov 03 20:50:01.757409 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:01.757377 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7bb9489676-z6z42" event={"ID":"b6725e63-2f8d-4f1b-88ad-41057dd4a5c2","Type":"ContainerStarted","Data":"ed98503402cf62f514e1e44ae5efb1cdded59de138158c53c3af49e78e5d60de"} Nov 03 20:50:01.757878 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:01.757481 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-7bb9489676-z6z42" Nov 03 20:50:01.758986 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:01.758963 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-dctzh" event={"ID":"d4d05666-5bea-415a-b266-805722b1158f","Type":"ContainerStarted","Data":"42401c318f6db6721b4b9e2d0176fc7d120861aa124a900785d7a46a480bc19e"} Nov 03 20:50:01.759113 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:01.759095 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-dctzh" Nov 03 20:50:01.874482 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:01.874415 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-7bb9489676-z6z42" podStartSLOduration=2.086145958 podStartE2EDuration="4.874401597s" totalImagesPullingTime="2.788255639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:49:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:50:01.872106278 +0000 UTC m=+637.097426091" watchObservedRunningTime="2026-11-03 20:50:01.874401597 +0000 UTC m=+637.099721387" Nov 03 20:50:01.991400 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:01.991353 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-dctzh" podStartSLOduration=2.090269459 podStartE2EDuration="4.991338514s" totalImagesPullingTime="2.901069055s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:49:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:50:01.986991937 +0000 UTC m=+637.212311753" watchObservedRunningTime="2026-11-03 20:50:01.991338514 +0000 UTC m=+637.216658303" Nov 03 20:50:04.654221 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:04.654187 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-6dc979975f-xfdzl"] Nov 03 20:50:04.657845 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:04.657825 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6dc979975f-xfdzl" Nov 03 20:50:04.710278 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:04.710249 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 03 20:50:04.734552 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:04.734514 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56e52b0e-8463-429a-8a83-6ce4d657d6e6-metrics-certs\") pod \"ai-gateway-controller-6dc979975f-xfdzl\" (UID: \"56e52b0e-8463-429a-8a83-6ce4d657d6e6\") " pod="opendatahub/ai-gateway-controller-6dc979975f-xfdzl" Nov 03 20:50:04.734720 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:04.734694 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h55xd\" (UniqueName: \"kubernetes.io/projected/56e52b0e-8463-429a-8a83-6ce4d657d6e6-kube-api-access-h55xd\") pod \"ai-gateway-controller-6dc979975f-xfdzl\" (UID: \"56e52b0e-8463-429a-8a83-6ce4d657d6e6\") " pod="opendatahub/ai-gateway-controller-6dc979975f-xfdzl" Nov 03 20:50:04.759555 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:04.759529 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-twgz9\"" Nov 03 20:50:04.759725 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:04.759622 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 03 20:50:04.761581 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:04.761563 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6dc979975f-xfdzl"] Nov 03 20:50:04.835444 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:04.835396 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56e52b0e-8463-429a-8a83-6ce4d657d6e6-metrics-certs\") pod \"ai-gateway-controller-6dc979975f-xfdzl\" (UID: \"56e52b0e-8463-429a-8a83-6ce4d657d6e6\") " pod="opendatahub/ai-gateway-controller-6dc979975f-xfdzl" Nov 03 20:50:04.835648 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:04.835519 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h55xd\" (UniqueName: \"kubernetes.io/projected/56e52b0e-8463-429a-8a83-6ce4d657d6e6-kube-api-access-h55xd\") pod \"ai-gateway-controller-6dc979975f-xfdzl\" (UID: \"56e52b0e-8463-429a-8a83-6ce4d657d6e6\") " pod="opendatahub/ai-gateway-controller-6dc979975f-xfdzl" Nov 03 20:50:04.838079 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:04.838057 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56e52b0e-8463-429a-8a83-6ce4d657d6e6-metrics-certs\") pod \"ai-gateway-controller-6dc979975f-xfdzl\" (UID: \"56e52b0e-8463-429a-8a83-6ce4d657d6e6\") " pod="opendatahub/ai-gateway-controller-6dc979975f-xfdzl" Nov 03 20:50:04.897245 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:04.897220 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h55xd\" (UniqueName: \"kubernetes.io/projected/56e52b0e-8463-429a-8a83-6ce4d657d6e6-kube-api-access-h55xd\") pod \"ai-gateway-controller-6dc979975f-xfdzl\" (UID: \"56e52b0e-8463-429a-8a83-6ce4d657d6e6\") " pod="opendatahub/ai-gateway-controller-6dc979975f-xfdzl" Nov 03 20:50:04.971138 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:04.971037 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6dc979975f-xfdzl" Nov 03 20:50:05.343640 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:05.343612 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6dc979975f-xfdzl"] Nov 03 20:50:05.345779 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:50:05.345743 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56e52b0e_8463_429a_8a83_6ce4d657d6e6.slice/crio-8cf1e66df7dee87b2d2365619f39c4ba95453ca9dca0c92e053524280e7b6149 WatchSource:0}: Error finding container 8cf1e66df7dee87b2d2365619f39c4ba95453ca9dca0c92e053524280e7b6149: Status 404 returned error can't find the container with id 8cf1e66df7dee87b2d2365619f39c4ba95453ca9dca0c92e053524280e7b6149 Nov 03 20:50:05.777376 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:05.777332 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6dc979975f-xfdzl" event={"ID":"56e52b0e-8463-429a-8a83-6ce4d657d6e6","Type":"ContainerStarted","Data":"8cf1e66df7dee87b2d2365619f39c4ba95453ca9dca0c92e053524280e7b6149"} Nov 03 20:50:08.795313 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:08.795278 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6dc979975f-xfdzl" event={"ID":"56e52b0e-8463-429a-8a83-6ce4d657d6e6","Type":"ContainerStarted","Data":"7862301ca5b12e8c230349f750df51718d7de93fb574fbdb2f633c74c69e52e5"} Nov 03 20:50:08.795773 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:08.795349 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-6dc979975f-xfdzl" Nov 03 20:50:08.911923 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:08.911848 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-6dc979975f-xfdzl" podStartSLOduration=2.087625093 podStartE2EDuration="4.911823016s" totalImagesPullingTime="2.824197923s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:50:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:50:08.910571509 +0000 UTC m=+644.135891308" watchObservedRunningTime="2026-11-03 20:50:08.911823016 +0000 UTC m=+644.137142806" Nov 03 20:50:12.766731 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:12.766706 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-7bb9489676-z6z42" Nov 03 20:50:12.767129 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:12.766771 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-dctzh" Nov 03 20:50:16.321079 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:50:16.321037 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 20:50:16.743643 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:16.743609 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7"] Nov 03 20:50:16.747118 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:16.747098 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7" Nov 03 20:50:16.860278 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:16.860236 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39ce2073-d106-4204-be5d-9ea93f510db8-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7\" (UID: \"39ce2073-d106-4204-be5d-9ea93f510db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7" Nov 03 20:50:16.860501 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:16.860344 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39ce2073-d106-4204-be5d-9ea93f510db8-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7\" (UID: \"39ce2073-d106-4204-be5d-9ea93f510db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7" Nov 03 20:50:16.860501 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:16.860390 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39ce2073-d106-4204-be5d-9ea93f510db8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7\" (UID: \"39ce2073-d106-4204-be5d-9ea93f510db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7" Nov 03 20:50:16.860501 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:16.860424 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39ce2073-d106-4204-be5d-9ea93f510db8-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7\" (UID: \"39ce2073-d106-4204-be5d-9ea93f510db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7" Nov 03 20:50:16.860630 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:16.860533 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39ce2073-d106-4204-be5d-9ea93f510db8-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7\" (UID: \"39ce2073-d106-4204-be5d-9ea93f510db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7" Nov 03 20:50:16.860670 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:16.860639 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39ce2073-d106-4204-be5d-9ea93f510db8-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7\" (UID: \"39ce2073-d106-4204-be5d-9ea93f510db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7" Nov 03 20:50:16.860707 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:16.860668 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gqcs\" (UniqueName: \"kubernetes.io/projected/39ce2073-d106-4204-be5d-9ea93f510db8-kube-api-access-2gqcs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7\" (UID: \"39ce2073-d106-4204-be5d-9ea93f510db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7" Nov 03 20:50:16.860707 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:16.860697 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39ce2073-d106-4204-be5d-9ea93f510db8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7\" (UID: \"39ce2073-d106-4204-be5d-9ea93f510db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7" Nov 03 20:50:16.860843 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:16.860828 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39ce2073-d106-4204-be5d-9ea93f510db8-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7\" (UID: \"39ce2073-d106-4204-be5d-9ea93f510db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7" Nov 03 20:50:16.954724 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:16.954692 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7"] Nov 03 20:50:16.961403 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:16.961374 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39ce2073-d106-4204-be5d-9ea93f510db8-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7\" (UID: \"39ce2073-d106-4204-be5d-9ea93f510db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7" Nov 03 20:50:16.961585 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:16.961415 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39ce2073-d106-4204-be5d-9ea93f510db8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7\" (UID: \"39ce2073-d106-4204-be5d-9ea93f510db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7" Nov 03 20:50:16.961585 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:16.961566 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39ce2073-d106-4204-be5d-9ea93f510db8-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7\" (UID: \"39ce2073-d106-4204-be5d-9ea93f510db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7" Nov 03 20:50:16.961694 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:16.961619 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39ce2073-d106-4204-be5d-9ea93f510db8-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7\" (UID: \"39ce2073-d106-4204-be5d-9ea93f510db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7" Nov 03 20:50:16.961749 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:16.961694 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39ce2073-d106-4204-be5d-9ea93f510db8-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7\" (UID: \"39ce2073-d106-4204-be5d-9ea93f510db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7" Nov 03 20:50:16.961749 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:16.961721 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gqcs\" (UniqueName: \"kubernetes.io/projected/39ce2073-d106-4204-be5d-9ea93f510db8-kube-api-access-2gqcs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7\" (UID: \"39ce2073-d106-4204-be5d-9ea93f510db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7" Nov 03 20:50:16.961921 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:16.961754 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39ce2073-d106-4204-be5d-9ea93f510db8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7\" (UID: \"39ce2073-d106-4204-be5d-9ea93f510db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7" Nov 03 20:50:16.961921 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:16.961812 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39ce2073-d106-4204-be5d-9ea93f510db8-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7\" (UID: \"39ce2073-d106-4204-be5d-9ea93f510db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7" Nov 03 20:50:16.961921 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:16.961822 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39ce2073-d106-4204-be5d-9ea93f510db8-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7\" (UID: \"39ce2073-d106-4204-be5d-9ea93f510db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7" Nov 03 20:50:16.962071 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:16.961918 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39ce2073-d106-4204-be5d-9ea93f510db8-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7\" (UID: \"39ce2073-d106-4204-be5d-9ea93f510db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7" Nov 03 20:50:16.962071 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:16.961975 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39ce2073-d106-4204-be5d-9ea93f510db8-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7\" (UID: \"39ce2073-d106-4204-be5d-9ea93f510db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7" Nov 03 20:50:16.962071 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:16.961996 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39ce2073-d106-4204-be5d-9ea93f510db8-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7\" (UID: \"39ce2073-d106-4204-be5d-9ea93f510db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7" Nov 03 20:50:16.962071 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:16.962050 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39ce2073-d106-4204-be5d-9ea93f510db8-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7\" (UID: \"39ce2073-d106-4204-be5d-9ea93f510db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7" Nov 03 20:50:16.962374 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:16.962354 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39ce2073-d106-4204-be5d-9ea93f510db8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7\" (UID: \"39ce2073-d106-4204-be5d-9ea93f510db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7" Nov 03 20:50:16.964639 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:16.964609 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39ce2073-d106-4204-be5d-9ea93f510db8-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7\" (UID: \"39ce2073-d106-4204-be5d-9ea93f510db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7" Nov 03 20:50:16.964777 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:16.964750 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39ce2073-d106-4204-be5d-9ea93f510db8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7\" (UID: \"39ce2073-d106-4204-be5d-9ea93f510db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7" Nov 03 20:50:17.022782 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:17.022709 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39ce2073-d106-4204-be5d-9ea93f510db8-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7\" (UID: \"39ce2073-d106-4204-be5d-9ea93f510db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7" Nov 03 20:50:17.023270 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:17.023251 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gqcs\" (UniqueName: \"kubernetes.io/projected/39ce2073-d106-4204-be5d-9ea93f510db8-kube-api-access-2gqcs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7\" (UID: \"39ce2073-d106-4204-be5d-9ea93f510db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7" Nov 03 20:50:17.059296 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:17.059263 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7" Nov 03 20:50:17.487711 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:17.487688 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7"] Nov 03 20:50:17.490285 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:50:17.490251 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39ce2073_d106_4204_be5d_9ea93f510db8.slice/crio-521fd60f5cdf8a032a8fd0e3aeed9fb7dee62b1cbd7f6610b378de2e51d9d36c WatchSource:0}: Error finding container 521fd60f5cdf8a032a8fd0e3aeed9fb7dee62b1cbd7f6610b378de2e51d9d36c: Status 404 returned error can't find the container with id 521fd60f5cdf8a032a8fd0e3aeed9fb7dee62b1cbd7f6610b378de2e51d9d36c Nov 03 20:50:17.492666 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:17.492635 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:50:17.492742 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:17.492701 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:50:17.492742 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:17.492731 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:50:17.842419 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:17.842337 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7" event={"ID":"39ce2073-d106-4204-be5d-9ea93f510db8","Type":"ContainerStarted","Data":"95f0c87e9c27adfd0113034e263a9cdd1a09ae70aee578bb2728fe2769424624"} Nov 03 20:50:17.842419 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:17.842384 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7" event={"ID":"39ce2073-d106-4204-be5d-9ea93f510db8","Type":"ContainerStarted","Data":"521fd60f5cdf8a032a8fd0e3aeed9fb7dee62b1cbd7f6610b378de2e51d9d36c"} Nov 03 20:50:17.963978 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:17.963917 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7" podStartSLOduration=1.963901353 podStartE2EDuration="1.963901353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:50:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:50:17.96257121 +0000 UTC m=+653.187891001" watchObservedRunningTime="2026-11-03 20:50:17.963901353 +0000 UTC m=+653.189221143" Nov 03 20:50:18.064458 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:18.062521 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7" Nov 03 20:50:18.064458 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:18.062669 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 03 20:50:18.064458 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:18.062718 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7" podUID="39ce2073-d106-4204-be5d-9ea93f510db8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 03 20:50:19.060929 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:19.060891 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 03 20:50:19.061616 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:19.061572 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7" podUID="39ce2073-d106-4204-be5d-9ea93f510db8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 03 20:50:19.802900 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:19.802863 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-6dc979975f-xfdzl" Nov 03 20:50:20.076553 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:20.060925 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 03 20:50:20.076972 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:20.076577 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7" podUID="39ce2073-d106-4204-be5d-9ea93f510db8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 03 20:50:21.068900 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:21.068860 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 03 20:50:21.069239 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:21.069207 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7" podUID="39ce2073-d106-4204-be5d-9ea93f510db8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 03 20:50:22.067807 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:22.067771 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 03 20:50:22.068451 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:22.068399 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7" podUID="39ce2073-d106-4204-be5d-9ea93f510db8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 03 20:50:23.070460 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:23.064821 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 03 20:50:23.070460 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:23.064882 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7" podUID="39ce2073-d106-4204-be5d-9ea93f510db8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 03 20:50:24.060569 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:24.060534 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 03 20:50:24.060744 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:24.060590 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7" podUID="39ce2073-d106-4204-be5d-9ea93f510db8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 03 20:50:25.060188 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:25.060144 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 03 20:50:25.060625 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:25.060215 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7" podUID="39ce2073-d106-4204-be5d-9ea93f510db8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 03 20:50:26.063721 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:26.063691 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7" Nov 03 20:50:26.064226 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:26.064155 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7" Nov 03 20:50:26.065007 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:50:26.064976 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7" Nov 03 20:50:46.321273 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:50:46.321185 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 20:51:00.716760 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:00.716719 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft"] Nov 03 20:51:00.719460 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:00.719424 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft" Nov 03 20:51:00.774363 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:00.774333 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-ca4642-openshift-default-dockercfg-gbv98\"" Nov 03 20:51:00.825189 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:00.825150 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft"] Nov 03 20:51:00.908533 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:00.908491 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/365e3804-37c5-4398-95c0-a1602237453e-istio-podinfo\") pod \"e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft\" (UID: \"365e3804-37c5-4398-95c0-a1602237453e\") " pod="openshift-ingress/e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft" Nov 03 20:51:00.908533 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:00.908529 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/365e3804-37c5-4398-95c0-a1602237453e-istio-data\") pod \"e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft\" (UID: \"365e3804-37c5-4398-95c0-a1602237453e\") " pod="openshift-ingress/e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft" Nov 03 20:51:00.908784 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:00.908548 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/365e3804-37c5-4398-95c0-a1602237453e-istio-envoy\") pod \"e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft\" (UID: \"365e3804-37c5-4398-95c0-a1602237453e\") " pod="openshift-ingress/e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft" Nov 03 20:51:00.908784 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:00.908664 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/365e3804-37c5-4398-95c0-a1602237453e-workload-socket\") pod \"e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft\" (UID: \"365e3804-37c5-4398-95c0-a1602237453e\") " pod="openshift-ingress/e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft" Nov 03 20:51:00.908784 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:00.908715 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8ps4\" (UniqueName: \"kubernetes.io/projected/365e3804-37c5-4398-95c0-a1602237453e-kube-api-access-c8ps4\") pod \"e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft\" (UID: \"365e3804-37c5-4398-95c0-a1602237453e\") " pod="openshift-ingress/e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft" Nov 03 20:51:00.908952 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:00.908845 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/365e3804-37c5-4398-95c0-a1602237453e-credential-socket\") pod \"e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft\" (UID: \"365e3804-37c5-4398-95c0-a1602237453e\") " pod="openshift-ingress/e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft" Nov 03 20:51:00.908952 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:00.908896 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/365e3804-37c5-4398-95c0-a1602237453e-istiod-ca-cert\") pod \"e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft\" (UID: \"365e3804-37c5-4398-95c0-a1602237453e\") " pod="openshift-ingress/e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft" Nov 03 20:51:00.909075 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:00.908977 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/365e3804-37c5-4398-95c0-a1602237453e-workload-certs\") pod \"e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft\" (UID: \"365e3804-37c5-4398-95c0-a1602237453e\") " pod="openshift-ingress/e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft" Nov 03 20:51:00.909075 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:00.909024 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/365e3804-37c5-4398-95c0-a1602237453e-istio-token\") pod \"e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft\" (UID: \"365e3804-37c5-4398-95c0-a1602237453e\") " pod="openshift-ingress/e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft" Nov 03 20:51:01.010149 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:01.010064 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/365e3804-37c5-4398-95c0-a1602237453e-workload-socket\") pod \"e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft\" (UID: \"365e3804-37c5-4398-95c0-a1602237453e\") " pod="openshift-ingress/e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft" Nov 03 20:51:01.010149 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:01.010125 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8ps4\" (UniqueName: \"kubernetes.io/projected/365e3804-37c5-4398-95c0-a1602237453e-kube-api-access-c8ps4\") pod \"e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft\" (UID: \"365e3804-37c5-4398-95c0-a1602237453e\") " pod="openshift-ingress/e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft" Nov 03 20:51:01.010379 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:01.010192 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/365e3804-37c5-4398-95c0-a1602237453e-credential-socket\") pod \"e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft\" (UID: \"365e3804-37c5-4398-95c0-a1602237453e\") " pod="openshift-ingress/e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft" Nov 03 20:51:01.010379 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:01.010230 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/365e3804-37c5-4398-95c0-a1602237453e-istiod-ca-cert\") pod \"e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft\" (UID: \"365e3804-37c5-4398-95c0-a1602237453e\") " pod="openshift-ingress/e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft" Nov 03 20:51:01.010379 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:01.010265 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/365e3804-37c5-4398-95c0-a1602237453e-workload-certs\") pod \"e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft\" (UID: \"365e3804-37c5-4398-95c0-a1602237453e\") " pod="openshift-ingress/e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft" Nov 03 20:51:01.010574 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:01.010411 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/365e3804-37c5-4398-95c0-a1602237453e-istio-token\") pod \"e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft\" (UID: \"365e3804-37c5-4398-95c0-a1602237453e\") " pod="openshift-ingress/e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft" Nov 03 20:51:01.010574 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:01.010544 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/365e3804-37c5-4398-95c0-a1602237453e-istio-podinfo\") pod \"e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft\" (UID: \"365e3804-37c5-4398-95c0-a1602237453e\") " pod="openshift-ingress/e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft" Nov 03 20:51:01.010698 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:01.010571 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/365e3804-37c5-4398-95c0-a1602237453e-workload-socket\") pod \"e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft\" (UID: \"365e3804-37c5-4398-95c0-a1602237453e\") " pod="openshift-ingress/e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft" Nov 03 20:51:01.010698 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:01.010577 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/365e3804-37c5-4398-95c0-a1602237453e-istio-data\") pod \"e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft\" (UID: \"365e3804-37c5-4398-95c0-a1602237453e\") " pod="openshift-ingress/e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft" Nov 03 20:51:01.010698 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:01.010639 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/365e3804-37c5-4398-95c0-a1602237453e-credential-socket\") pod \"e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft\" (UID: \"365e3804-37c5-4398-95c0-a1602237453e\") " pod="openshift-ingress/e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft" Nov 03 20:51:01.010698 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:01.010668 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/365e3804-37c5-4398-95c0-a1602237453e-istio-envoy\") pod \"e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft\" (UID: \"365e3804-37c5-4398-95c0-a1602237453e\") " pod="openshift-ingress/e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft" Nov 03 20:51:01.010698 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:01.010695 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/365e3804-37c5-4398-95c0-a1602237453e-workload-certs\") pod \"e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft\" (UID: \"365e3804-37c5-4398-95c0-a1602237453e\") " pod="openshift-ingress/e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft" Nov 03 20:51:01.010975 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:01.010824 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/365e3804-37c5-4398-95c0-a1602237453e-istio-data\") pod \"e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft\" (UID: \"365e3804-37c5-4398-95c0-a1602237453e\") " pod="openshift-ingress/e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft" Nov 03 20:51:01.011104 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:01.011079 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/365e3804-37c5-4398-95c0-a1602237453e-istiod-ca-cert\") pod \"e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft\" (UID: \"365e3804-37c5-4398-95c0-a1602237453e\") " pod="openshift-ingress/e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft" Nov 03 20:51:01.013275 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:01.013252 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/365e3804-37c5-4398-95c0-a1602237453e-istio-envoy\") pod \"e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft\" (UID: \"365e3804-37c5-4398-95c0-a1602237453e\") " pod="openshift-ingress/e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft" Nov 03 20:51:01.013514 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:01.013498 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/365e3804-37c5-4398-95c0-a1602237453e-istio-podinfo\") pod \"e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft\" (UID: \"365e3804-37c5-4398-95c0-a1602237453e\") " pod="openshift-ingress/e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft" Nov 03 20:51:01.071662 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:01.071636 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/365e3804-37c5-4398-95c0-a1602237453e-istio-token\") pod \"e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft\" (UID: \"365e3804-37c5-4398-95c0-a1602237453e\") " pod="openshift-ingress/e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft" Nov 03 20:51:01.072115 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:01.072092 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8ps4\" (UniqueName: \"kubernetes.io/projected/365e3804-37c5-4398-95c0-a1602237453e-kube-api-access-c8ps4\") pod \"e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft\" (UID: \"365e3804-37c5-4398-95c0-a1602237453e\") " pod="openshift-ingress/e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft" Nov 03 20:51:01.331924 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:01.331811 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft" Nov 03 20:51:01.355758 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:01.355729 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28"] Nov 03 20:51:01.358924 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:01.358902 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28" Nov 03 20:51:01.411343 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:01.411309 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-89bb58-openshift-default-dockercfg-26hlr\"" Nov 03 20:51:01.420945 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:01.420731 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28"] Nov 03 20:51:01.516502 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:01.516464 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58829723-0628-457e-80b8-f81d15ecfe2e-istio-data\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28\" (UID: \"58829723-0628-457e-80b8-f81d15ecfe2e\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28" Nov 03 20:51:01.516713 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:01.516607 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58829723-0628-457e-80b8-f81d15ecfe2e-workload-socket\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28\" (UID: \"58829723-0628-457e-80b8-f81d15ecfe2e\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28" Nov 03 20:51:01.516713 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:01.516654 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58829723-0628-457e-80b8-f81d15ecfe2e-istio-token\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28\" (UID: \"58829723-0628-457e-80b8-f81d15ecfe2e\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28" Nov 03 20:51:01.516845 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:01.516807 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58829723-0628-457e-80b8-f81d15ecfe2e-istio-envoy\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28\" (UID: \"58829723-0628-457e-80b8-f81d15ecfe2e\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28" Nov 03 20:51:01.516900 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:01.516853 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58829723-0628-457e-80b8-f81d15ecfe2e-workload-certs\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28\" (UID: \"58829723-0628-457e-80b8-f81d15ecfe2e\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28" Nov 03 20:51:01.516954 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:01.516897 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58829723-0628-457e-80b8-f81d15ecfe2e-istiod-ca-cert\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28\" (UID: \"58829723-0628-457e-80b8-f81d15ecfe2e\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28" Nov 03 20:51:01.517310 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:01.517233 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58829723-0628-457e-80b8-f81d15ecfe2e-credential-socket\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28\" (UID: \"58829723-0628-457e-80b8-f81d15ecfe2e\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28" Nov 03 20:51:01.517310 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:01.517275 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9bmc\" (UniqueName: \"kubernetes.io/projected/58829723-0628-457e-80b8-f81d15ecfe2e-kube-api-access-l9bmc\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28\" (UID: \"58829723-0628-457e-80b8-f81d15ecfe2e\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28" Nov 03 20:51:01.517682 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:01.517629 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58829723-0628-457e-80b8-f81d15ecfe2e-istio-podinfo\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28\" (UID: \"58829723-0628-457e-80b8-f81d15ecfe2e\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28" Nov 03 20:51:01.618978 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:01.618898 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58829723-0628-457e-80b8-f81d15ecfe2e-credential-socket\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28\" (UID: \"58829723-0628-457e-80b8-f81d15ecfe2e\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28" Nov 03 20:51:01.618978 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:01.618933 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9bmc\" (UniqueName: \"kubernetes.io/projected/58829723-0628-457e-80b8-f81d15ecfe2e-kube-api-access-l9bmc\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28\" (UID: \"58829723-0628-457e-80b8-f81d15ecfe2e\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28" Nov 03 20:51:01.618978 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:01.618955 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58829723-0628-457e-80b8-f81d15ecfe2e-istio-podinfo\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28\" (UID: \"58829723-0628-457e-80b8-f81d15ecfe2e\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28" Nov 03 20:51:01.619261 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:01.619015 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58829723-0628-457e-80b8-f81d15ecfe2e-istio-data\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28\" (UID: \"58829723-0628-457e-80b8-f81d15ecfe2e\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28" Nov 03 20:51:01.619261 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:01.619042 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58829723-0628-457e-80b8-f81d15ecfe2e-workload-socket\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28\" (UID: \"58829723-0628-457e-80b8-f81d15ecfe2e\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28" Nov 03 20:51:01.619261 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:01.619060 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58829723-0628-457e-80b8-f81d15ecfe2e-istio-token\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28\" (UID: \"58829723-0628-457e-80b8-f81d15ecfe2e\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28" Nov 03 20:51:01.619261 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:01.619134 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58829723-0628-457e-80b8-f81d15ecfe2e-istio-envoy\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28\" (UID: \"58829723-0628-457e-80b8-f81d15ecfe2e\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28" Nov 03 20:51:01.619261 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:01.619162 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58829723-0628-457e-80b8-f81d15ecfe2e-workload-certs\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28\" (UID: \"58829723-0628-457e-80b8-f81d15ecfe2e\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28" Nov 03 20:51:01.619261 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:01.619184 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58829723-0628-457e-80b8-f81d15ecfe2e-istiod-ca-cert\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28\" (UID: \"58829723-0628-457e-80b8-f81d15ecfe2e\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28" Nov 03 20:51:01.619674 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:01.619364 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58829723-0628-457e-80b8-f81d15ecfe2e-credential-socket\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28\" (UID: \"58829723-0628-457e-80b8-f81d15ecfe2e\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28" Nov 03 20:51:01.619674 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:01.619416 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58829723-0628-457e-80b8-f81d15ecfe2e-istio-data\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28\" (UID: \"58829723-0628-457e-80b8-f81d15ecfe2e\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28" Nov 03 20:51:01.619674 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:01.619529 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58829723-0628-457e-80b8-f81d15ecfe2e-workload-socket\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28\" (UID: \"58829723-0628-457e-80b8-f81d15ecfe2e\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28" Nov 03 20:51:01.619816 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:01.619696 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58829723-0628-457e-80b8-f81d15ecfe2e-workload-certs\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28\" (UID: \"58829723-0628-457e-80b8-f81d15ecfe2e\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28" Nov 03 20:51:01.619924 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:01.619904 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58829723-0628-457e-80b8-f81d15ecfe2e-istiod-ca-cert\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28\" (UID: \"58829723-0628-457e-80b8-f81d15ecfe2e\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28" Nov 03 20:51:01.621641 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:01.621623 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58829723-0628-457e-80b8-f81d15ecfe2e-istio-podinfo\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28\" (UID: \"58829723-0628-457e-80b8-f81d15ecfe2e\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28" Nov 03 20:51:01.621800 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:01.621785 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58829723-0628-457e-80b8-f81d15ecfe2e-istio-envoy\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28\" (UID: \"58829723-0628-457e-80b8-f81d15ecfe2e\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28" Nov 03 20:51:01.682899 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:01.682864 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58829723-0628-457e-80b8-f81d15ecfe2e-istio-token\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28\" (UID: \"58829723-0628-457e-80b8-f81d15ecfe2e\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28" Nov 03 20:51:01.683052 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:01.683011 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9bmc\" (UniqueName: \"kubernetes.io/projected/58829723-0628-457e-80b8-f81d15ecfe2e-kube-api-access-l9bmc\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28\" (UID: \"58829723-0628-457e-80b8-f81d15ecfe2e\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28" Nov 03 20:51:01.768692 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:01.768655 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft"] Nov 03 20:51:01.770642 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:51:01.770615 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod365e3804_37c5_4398_95c0_a1602237453e.slice/crio-fa88a7604d023ddbf3be4ca2411a0f05afc49cda9bec12a206ee10f57e94bf67 WatchSource:0}: Error finding container fa88a7604d023ddbf3be4ca2411a0f05afc49cda9bec12a206ee10f57e94bf67: Status 404 returned error can't find the container with id fa88a7604d023ddbf3be4ca2411a0f05afc49cda9bec12a206ee10f57e94bf67 Nov 03 20:51:01.773048 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:01.773018 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:51:01.773124 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:01.773082 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:51:01.773124 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:01.773106 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:51:01.971724 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:01.971694 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28" Nov 03 20:51:02.080852 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:02.080806 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft" event={"ID":"365e3804-37c5-4398-95c0-a1602237453e","Type":"ContainerStarted","Data":"8c843261fb8ad8e8ef30b9bc672917ec5843dfc9995c59389d3b8fe006d9a0f2"} Nov 03 20:51:02.081034 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:02.080874 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft" event={"ID":"365e3804-37c5-4398-95c0-a1602237453e","Type":"ContainerStarted","Data":"fa88a7604d023ddbf3be4ca2411a0f05afc49cda9bec12a206ee10f57e94bf67"} Nov 03 20:51:02.201737 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:02.201672 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft" podStartSLOduration=2.201647292 podStartE2EDuration="2.201647292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:02.199626081 +0000 UTC m=+697.424945869" watchObservedRunningTime="2026-11-03 20:51:02.201647292 +0000 UTC m=+697.426967081" Nov 03 20:51:02.332380 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:02.332284 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft" Nov 03 20:51:02.338354 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:02.338324 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft" Nov 03 20:51:02.421780 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:02.421735 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28"] Nov 03 20:51:02.422496 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:51:02.422470 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58829723_0628_457e_80b8_f81d15ecfe2e.slice/crio-a5e133206cfceedb5ba0470db9c881852cfb9e7ee52dee7732c7be09df305869 WatchSource:0}: Error finding container a5e133206cfceedb5ba0470db9c881852cfb9e7ee52dee7732c7be09df305869: Status 404 returned error can't find the container with id a5e133206cfceedb5ba0470db9c881852cfb9e7ee52dee7732c7be09df305869 Nov 03 20:51:02.425639 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:02.425602 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:51:02.425735 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:02.425689 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:51:02.425803 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:02.425732 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:51:02.658268 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:02.658229 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p"] Nov 03 20:51:02.661237 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:02.661218 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p" Nov 03 20:51:02.758093 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:02.757698 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-d01cef-openshift-default-dockercfg-bxtfs\"" Nov 03 20:51:02.761079 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:02.761044 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p"] Nov 03 20:51:02.831771 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:02.831718 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4b14774-55f2-4fad-88ad-74d08bf328f7-istio-envoy\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p\" (UID: \"e4b14774-55f2-4fad-88ad-74d08bf328f7\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p" Nov 03 20:51:02.832290 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:02.831790 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4b14774-55f2-4fad-88ad-74d08bf328f7-istio-podinfo\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p\" (UID: \"e4b14774-55f2-4fad-88ad-74d08bf328f7\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p" Nov 03 20:51:02.832290 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:02.831851 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b14774-55f2-4fad-88ad-74d08bf328f7-credential-socket\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p\" (UID: \"e4b14774-55f2-4fad-88ad-74d08bf328f7\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p" Nov 03 20:51:02.832290 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:02.831944 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b14774-55f2-4fad-88ad-74d08bf328f7-workload-socket\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p\" (UID: \"e4b14774-55f2-4fad-88ad-74d08bf328f7\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p" Nov 03 20:51:02.832290 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:02.832054 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4b14774-55f2-4fad-88ad-74d08bf328f7-workload-certs\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p\" (UID: \"e4b14774-55f2-4fad-88ad-74d08bf328f7\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p" Nov 03 20:51:02.832290 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:02.832079 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4b14774-55f2-4fad-88ad-74d08bf328f7-istiod-ca-cert\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p\" (UID: \"e4b14774-55f2-4fad-88ad-74d08bf328f7\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p" Nov 03 20:51:02.832290 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:02.832106 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkvpg\" (UniqueName: \"kubernetes.io/projected/e4b14774-55f2-4fad-88ad-74d08bf328f7-kube-api-access-fkvpg\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p\" (UID: \"e4b14774-55f2-4fad-88ad-74d08bf328f7\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p" Nov 03 20:51:02.832290 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:02.832133 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4b14774-55f2-4fad-88ad-74d08bf328f7-istio-token\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p\" (UID: \"e4b14774-55f2-4fad-88ad-74d08bf328f7\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p" Nov 03 20:51:02.832290 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:02.832177 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4b14774-55f2-4fad-88ad-74d08bf328f7-istio-data\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p\" (UID: \"e4b14774-55f2-4fad-88ad-74d08bf328f7\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p" Nov 03 20:51:02.933891 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:02.933791 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b14774-55f2-4fad-88ad-74d08bf328f7-workload-socket\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p\" (UID: \"e4b14774-55f2-4fad-88ad-74d08bf328f7\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p" Nov 03 20:51:02.934076 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:02.933895 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4b14774-55f2-4fad-88ad-74d08bf328f7-workload-certs\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p\" (UID: \"e4b14774-55f2-4fad-88ad-74d08bf328f7\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p" Nov 03 20:51:02.934076 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:02.933924 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4b14774-55f2-4fad-88ad-74d08bf328f7-istiod-ca-cert\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p\" (UID: \"e4b14774-55f2-4fad-88ad-74d08bf328f7\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p" Nov 03 20:51:02.934076 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:02.933946 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkvpg\" (UniqueName: \"kubernetes.io/projected/e4b14774-55f2-4fad-88ad-74d08bf328f7-kube-api-access-fkvpg\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p\" (UID: \"e4b14774-55f2-4fad-88ad-74d08bf328f7\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p" Nov 03 20:51:02.934076 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:02.933972 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4b14774-55f2-4fad-88ad-74d08bf328f7-istio-token\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p\" (UID: \"e4b14774-55f2-4fad-88ad-74d08bf328f7\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p" Nov 03 20:51:02.934076 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:02.934007 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4b14774-55f2-4fad-88ad-74d08bf328f7-istio-data\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p\" (UID: \"e4b14774-55f2-4fad-88ad-74d08bf328f7\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p" Nov 03 20:51:02.934076 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:02.934065 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4b14774-55f2-4fad-88ad-74d08bf328f7-istio-envoy\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p\" (UID: \"e4b14774-55f2-4fad-88ad-74d08bf328f7\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p" Nov 03 20:51:02.934373 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:02.934098 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4b14774-55f2-4fad-88ad-74d08bf328f7-istio-podinfo\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p\" (UID: \"e4b14774-55f2-4fad-88ad-74d08bf328f7\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p" Nov 03 20:51:02.934373 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:02.934142 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b14774-55f2-4fad-88ad-74d08bf328f7-credential-socket\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p\" (UID: \"e4b14774-55f2-4fad-88ad-74d08bf328f7\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p" Nov 03 20:51:02.934373 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:02.934258 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b14774-55f2-4fad-88ad-74d08bf328f7-workload-socket\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p\" (UID: \"e4b14774-55f2-4fad-88ad-74d08bf328f7\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p" Nov 03 20:51:02.934566 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:02.934454 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4b14774-55f2-4fad-88ad-74d08bf328f7-workload-certs\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p\" (UID: \"e4b14774-55f2-4fad-88ad-74d08bf328f7\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p" Nov 03 20:51:02.934625 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:02.934597 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4b14774-55f2-4fad-88ad-74d08bf328f7-istio-data\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p\" (UID: \"e4b14774-55f2-4fad-88ad-74d08bf328f7\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p" Nov 03 20:51:02.934686 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:02.934650 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b14774-55f2-4fad-88ad-74d08bf328f7-credential-socket\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p\" (UID: \"e4b14774-55f2-4fad-88ad-74d08bf328f7\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p" Nov 03 20:51:02.934748 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:02.934724 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4b14774-55f2-4fad-88ad-74d08bf328f7-istiod-ca-cert\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p\" (UID: \"e4b14774-55f2-4fad-88ad-74d08bf328f7\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p" Nov 03 20:51:02.936575 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:02.936545 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4b14774-55f2-4fad-88ad-74d08bf328f7-istio-envoy\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p\" (UID: \"e4b14774-55f2-4fad-88ad-74d08bf328f7\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p" Nov 03 20:51:02.936806 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:02.936788 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4b14774-55f2-4fad-88ad-74d08bf328f7-istio-podinfo\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p\" (UID: \"e4b14774-55f2-4fad-88ad-74d08bf328f7\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p" Nov 03 20:51:02.995489 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:02.995453 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkvpg\" (UniqueName: \"kubernetes.io/projected/e4b14774-55f2-4fad-88ad-74d08bf328f7-kube-api-access-fkvpg\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p\" (UID: \"e4b14774-55f2-4fad-88ad-74d08bf328f7\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p" Nov 03 20:51:02.996932 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:02.996878 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4b14774-55f2-4fad-88ad-74d08bf328f7-istio-token\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p\" (UID: \"e4b14774-55f2-4fad-88ad-74d08bf328f7\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p" Nov 03 20:51:03.088146 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:03.088107 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28" event={"ID":"58829723-0628-457e-80b8-f81d15ecfe2e","Type":"ContainerStarted","Data":"10d10b551cde5c66d72ab000240f018c97e9ffa11aafc083cfab2f0a0f710fe3"} Nov 03 20:51:03.088335 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:03.088167 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28" event={"ID":"58829723-0628-457e-80b8-f81d15ecfe2e","Type":"ContainerStarted","Data":"a5e133206cfceedb5ba0470db9c881852cfb9e7ee52dee7732c7be09df305869"} Nov 03 20:51:03.088376 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:03.088337 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft" Nov 03 20:51:03.089337 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:03.089321 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft" Nov 03 20:51:03.210748 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:03.210639 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28" podStartSLOduration=2.210623394 podStartE2EDuration="2.210623394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:03.208311237 +0000 UTC m=+698.433631028" watchObservedRunningTime="2026-11-03 20:51:03.210623394 +0000 UTC m=+698.435943180" Nov 03 20:51:03.274473 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:03.274404 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p" Nov 03 20:51:03.724353 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:03.724328 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p"] Nov 03 20:51:03.726354 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:51:03.726324 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4b14774_55f2_4fad_88ad_74d08bf328f7.slice/crio-7a015d42e81af71e1aa9f387216ec123075116947fc600dc5d968b14e1558ca4 WatchSource:0}: Error finding container 7a015d42e81af71e1aa9f387216ec123075116947fc600dc5d968b14e1558ca4: Status 404 returned error can't find the container with id 7a015d42e81af71e1aa9f387216ec123075116947fc600dc5d968b14e1558ca4 Nov 03 20:51:03.728848 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:03.728812 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:51:03.728953 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:03.728888 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:51:03.728953 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:03.728917 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:51:03.971846 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:03.971800 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28" Nov 03 20:51:03.987036 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:03.986941 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28" Nov 03 20:51:04.098594 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:04.098563 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p" event={"ID":"e4b14774-55f2-4fad-88ad-74d08bf328f7","Type":"ContainerStarted","Data":"b25dde8b6a1675398d99debfe7fc44da9de0edb7814bcfe8cb24683fafbc35d3"} Nov 03 20:51:04.098769 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:04.098619 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p" event={"ID":"e4b14774-55f2-4fad-88ad-74d08bf328f7","Type":"ContainerStarted","Data":"7a015d42e81af71e1aa9f387216ec123075116947fc600dc5d968b14e1558ca4"} Nov 03 20:51:04.098879 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:04.098865 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28" Nov 03 20:51:04.099915 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:04.099895 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28" Nov 03 20:51:04.275077 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:04.274997 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p" Nov 03 20:51:04.280968 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:04.280939 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p" Nov 03 20:51:04.352656 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:04.352589 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p" podStartSLOduration=2.352569526 podStartE2EDuration="2.352569526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:04.351346855 +0000 UTC m=+699.576666673" watchObservedRunningTime="2026-11-03 20:51:04.352569526 +0000 UTC m=+699.577889320" Nov 03 20:51:05.103635 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:05.103596 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p" Nov 03 20:51:05.104642 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:05.104628 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p" Nov 03 20:51:11.171093 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:11.171052 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ca4642-56449c8659-vtm4r"] Nov 03 20:51:11.174789 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:11.174766 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ca4642-56449c8659-vtm4r" Nov 03 20:51:11.229093 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:11.229057 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-ca4642\"" Nov 03 20:51:11.241688 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:11.241659 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f951e66-9baa-4ad3-994f-fa1b2e0d4828-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-ca4642-56449c8659-vtm4r\" (UID: \"6f951e66-9baa-4ad3-994f-fa1b2e0d4828\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ca4642-56449c8659-vtm4r" Nov 03 20:51:11.241852 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:11.241715 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z75gs\" (UniqueName: \"kubernetes.io/projected/6f951e66-9baa-4ad3-994f-fa1b2e0d4828-kube-api-access-z75gs\") pod \"payload-pre-processing-e2e-worker-w4-ca4642-56449c8659-vtm4r\" (UID: \"6f951e66-9baa-4ad3-994f-fa1b2e0d4828\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ca4642-56449c8659-vtm4r" Nov 03 20:51:11.279676 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:11.279621 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-ca4642-dockercfg-9vzjm\"" Nov 03 20:51:11.286752 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:11.286714 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ca4642-56449c8659-vtm4r"] Nov 03 20:51:11.342963 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:11.342924 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f951e66-9baa-4ad3-994f-fa1b2e0d4828-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-ca4642-56449c8659-vtm4r\" (UID: \"6f951e66-9baa-4ad3-994f-fa1b2e0d4828\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ca4642-56449c8659-vtm4r" Nov 03 20:51:11.343145 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:11.342989 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z75gs\" (UniqueName: \"kubernetes.io/projected/6f951e66-9baa-4ad3-994f-fa1b2e0d4828-kube-api-access-z75gs\") pod \"payload-pre-processing-e2e-worker-w4-ca4642-56449c8659-vtm4r\" (UID: \"6f951e66-9baa-4ad3-994f-fa1b2e0d4828\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ca4642-56449c8659-vtm4r" Nov 03 20:51:11.343677 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:11.343659 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f951e66-9baa-4ad3-994f-fa1b2e0d4828-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-ca4642-56449c8659-vtm4r\" (UID: \"6f951e66-9baa-4ad3-994f-fa1b2e0d4828\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ca4642-56449c8659-vtm4r" Nov 03 20:51:11.403740 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:11.403681 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z75gs\" (UniqueName: \"kubernetes.io/projected/6f951e66-9baa-4ad3-994f-fa1b2e0d4828-kube-api-access-z75gs\") pod \"payload-pre-processing-e2e-worker-w4-ca4642-56449c8659-vtm4r\" (UID: \"6f951e66-9baa-4ad3-994f-fa1b2e0d4828\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ca4642-56449c8659-vtm4r" Nov 03 20:51:11.488497 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:11.488390 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ca4642-56449c8659-vtm4r" Nov 03 20:51:11.864817 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:11.864787 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ca4642-56449c8659-vtm4r"] Nov 03 20:51:11.866370 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:51:11.866339 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f951e66_9baa_4ad3_994f_fa1b2e0d4828.slice/crio-0d4f2adfcc0946dc8ef184d0bde60df4210634b8ff33c7dbb4293ae46f11e0f7 WatchSource:0}: Error finding container 0d4f2adfcc0946dc8ef184d0bde60df4210634b8ff33c7dbb4293ae46f11e0f7: Status 404 returned error can't find the container with id 0d4f2adfcc0946dc8ef184d0bde60df4210634b8ff33c7dbb4293ae46f11e0f7 Nov 03 20:51:12.139239 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:12.139155 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ca4642-56449c8659-vtm4r" event={"ID":"6f951e66-9baa-4ad3-994f-fa1b2e0d4828","Type":"ContainerStarted","Data":"4e05564f2fc434141b76871a062356686d18f7075903c6117bc78779d72f6c64"} Nov 03 20:51:12.139239 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:12.139203 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ca4642-56449c8659-vtm4r" event={"ID":"6f951e66-9baa-4ad3-994f-fa1b2e0d4828","Type":"ContainerStarted","Data":"0d4f2adfcc0946dc8ef184d0bde60df4210634b8ff33c7dbb4293ae46f11e0f7"} Nov 03 20:51:12.139460 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:12.139294 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ca4642-56449c8659-vtm4r" Nov 03 20:51:12.256004 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:12.255943 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ca4642-56449c8659-vtm4r" podStartSLOduration=1.255928578 podStartE2EDuration="1.255928578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:12.253739048 +0000 UTC m=+707.479058838" watchObservedRunningTime="2026-11-03 20:51:12.255928578 +0000 UTC m=+707.481248400" Nov 03 20:51:16.227095 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:16.227055 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-12a423-846d4b6867-grwcg"] Nov 03 20:51:16.231012 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:16.230989 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-12a423-846d4b6867-grwcg" Nov 03 20:51:16.305281 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:16.305250 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b0135ee-b547-46e8-9953-6a46cb61f2c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-12a423-846d4b6867-grwcg\" (UID: \"7b0135ee-b547-46e8-9953-6a46cb61f2c4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-12a423-846d4b6867-grwcg" Nov 03 20:51:16.305534 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:16.305471 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4h57\" (UniqueName: \"kubernetes.io/projected/7b0135ee-b547-46e8-9953-6a46cb61f2c4-kube-api-access-v4h57\") pod \"payload-pre-processing-e2e-worker-w6-12a423-846d4b6867-grwcg\" (UID: \"7b0135ee-b547-46e8-9953-6a46cb61f2c4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-12a423-846d4b6867-grwcg" Nov 03 20:51:16.322505 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:51:16.322473 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 20:51:16.356459 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:16.347177 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-12a423-dockercfg-p8lpr\"" Nov 03 20:51:16.356459 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:16.347288 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-12a423\"" Nov 03 20:51:16.361834 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:16.361789 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-12a423-846d4b6867-grwcg"] Nov 03 20:51:16.386900 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:16.386865 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-12a423-559b96f96d-nmf2b"] Nov 03 20:51:16.390688 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:16.390664 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-12a423-559b96f96d-nmf2b" Nov 03 20:51:16.406373 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:16.406350 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b0135ee-b547-46e8-9953-6a46cb61f2c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-12a423-846d4b6867-grwcg\" (UID: \"7b0135ee-b547-46e8-9953-6a46cb61f2c4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-12a423-846d4b6867-grwcg" Nov 03 20:51:16.406539 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:16.406450 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4h57\" (UniqueName: \"kubernetes.io/projected/7b0135ee-b547-46e8-9953-6a46cb61f2c4-kube-api-access-v4h57\") pod \"payload-pre-processing-e2e-worker-w6-12a423-846d4b6867-grwcg\" (UID: \"7b0135ee-b547-46e8-9953-6a46cb61f2c4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-12a423-846d4b6867-grwcg" Nov 03 20:51:16.407024 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:16.407000 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b0135ee-b547-46e8-9953-6a46cb61f2c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-12a423-846d4b6867-grwcg\" (UID: \"7b0135ee-b547-46e8-9953-6a46cb61f2c4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-12a423-846d4b6867-grwcg" Nov 03 20:51:16.421585 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:16.421559 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-12a423-559b96f96d-nmf2b"] Nov 03 20:51:16.467539 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:16.467514 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4h57\" (UniqueName: \"kubernetes.io/projected/7b0135ee-b547-46e8-9953-6a46cb61f2c4-kube-api-access-v4h57\") pod \"payload-pre-processing-e2e-worker-w6-12a423-846d4b6867-grwcg\" (UID: \"7b0135ee-b547-46e8-9953-6a46cb61f2c4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-12a423-846d4b6867-grwcg" Nov 03 20:51:16.507222 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:16.507135 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jtr2\" (UniqueName: \"kubernetes.io/projected/3e6b8d49-0db7-4bbc-9b3e-1105d2bdbc78-kube-api-access-4jtr2\") pod \"payload-processing-e2e-worker-w6-12a423-559b96f96d-nmf2b\" (UID: \"3e6b8d49-0db7-4bbc-9b3e-1105d2bdbc78\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-12a423-559b96f96d-nmf2b" Nov 03 20:51:16.507399 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:16.507319 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e6b8d49-0db7-4bbc-9b3e-1105d2bdbc78-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-12a423-559b96f96d-nmf2b\" (UID: \"3e6b8d49-0db7-4bbc-9b3e-1105d2bdbc78\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-12a423-559b96f96d-nmf2b" Nov 03 20:51:16.543889 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:16.543827 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-12a423-846d4b6867-grwcg" Nov 03 20:51:16.610367 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:16.610331 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e6b8d49-0db7-4bbc-9b3e-1105d2bdbc78-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-12a423-559b96f96d-nmf2b\" (UID: \"3e6b8d49-0db7-4bbc-9b3e-1105d2bdbc78\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-12a423-559b96f96d-nmf2b" Nov 03 20:51:16.610560 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:16.610504 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jtr2\" (UniqueName: \"kubernetes.io/projected/3e6b8d49-0db7-4bbc-9b3e-1105d2bdbc78-kube-api-access-4jtr2\") pod \"payload-processing-e2e-worker-w6-12a423-559b96f96d-nmf2b\" (UID: \"3e6b8d49-0db7-4bbc-9b3e-1105d2bdbc78\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-12a423-559b96f96d-nmf2b" Nov 03 20:51:16.612085 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:16.612052 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e6b8d49-0db7-4bbc-9b3e-1105d2bdbc78-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-12a423-559b96f96d-nmf2b\" (UID: \"3e6b8d49-0db7-4bbc-9b3e-1105d2bdbc78\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-12a423-559b96f96d-nmf2b" Nov 03 20:51:16.671619 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:16.671581 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jtr2\" (UniqueName: \"kubernetes.io/projected/3e6b8d49-0db7-4bbc-9b3e-1105d2bdbc78-kube-api-access-4jtr2\") pod \"payload-processing-e2e-worker-w6-12a423-559b96f96d-nmf2b\" (UID: \"3e6b8d49-0db7-4bbc-9b3e-1105d2bdbc78\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-12a423-559b96f96d-nmf2b" Nov 03 20:51:16.704603 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:16.704569 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-12a423-559b96f96d-nmf2b" Nov 03 20:51:16.934197 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:16.930928 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-12a423-846d4b6867-grwcg"] Nov 03 20:51:17.080223 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:17.080200 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-12a423-559b96f96d-nmf2b"] Nov 03 20:51:17.082350 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:51:17.082324 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e6b8d49_0db7_4bbc_9b3e_1105d2bdbc78.slice/crio-d64cd7e6e5dedffae22c1db7aab85e69beb880716678a18ff99d8a1a28e9d10d WatchSource:0}: Error finding container d64cd7e6e5dedffae22c1db7aab85e69beb880716678a18ff99d8a1a28e9d10d: Status 404 returned error can't find the container with id d64cd7e6e5dedffae22c1db7aab85e69beb880716678a18ff99d8a1a28e9d10d Nov 03 20:51:17.164637 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:17.164600 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-12a423-846d4b6867-grwcg" event={"ID":"7b0135ee-b547-46e8-9953-6a46cb61f2c4","Type":"ContainerStarted","Data":"e69e1c1980274f4b77b456df4c88e10de37015cb0a6d99dfa49f4a8f433dfd4c"} Nov 03 20:51:17.164823 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:17.164672 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-12a423-846d4b6867-grwcg" Nov 03 20:51:17.164823 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:17.164696 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-12a423-846d4b6867-grwcg" event={"ID":"7b0135ee-b547-46e8-9953-6a46cb61f2c4","Type":"ContainerStarted","Data":"5b57d66efcbd3ed9ffff0dcb6c4b92d686e0041ced2b2611ee08b4defc4b3c75"} Nov 03 20:51:17.166110 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:17.166081 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-12a423-559b96f96d-nmf2b" event={"ID":"3e6b8d49-0db7-4bbc-9b3e-1105d2bdbc78","Type":"ContainerStarted","Data":"3f479753f615643b9fe749dbdaeb5ddde4eace1984c7faab8edcedfef05c9ea3"} Nov 03 20:51:17.166209 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:17.166118 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-12a423-559b96f96d-nmf2b" event={"ID":"3e6b8d49-0db7-4bbc-9b3e-1105d2bdbc78","Type":"ContainerStarted","Data":"d64cd7e6e5dedffae22c1db7aab85e69beb880716678a18ff99d8a1a28e9d10d"} Nov 03 20:51:17.166209 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:17.166157 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-12a423-559b96f96d-nmf2b" Nov 03 20:51:17.282569 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:17.282522 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-12a423-846d4b6867-grwcg" podStartSLOduration=1.282506602 podStartE2EDuration="1.282506602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:17.280735985 +0000 UTC m=+712.506055787" watchObservedRunningTime="2026-11-03 20:51:17.282506602 +0000 UTC m=+712.507826391" Nov 03 20:51:17.408182 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:17.408128 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-12a423-559b96f96d-nmf2b" podStartSLOduration=1.408110648 podStartE2EDuration="1.408110648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:17.406380509 +0000 UTC m=+712.631700335" watchObservedRunningTime="2026-11-03 20:51:17.408110648 +0000 UTC m=+712.633430441" Nov 03 20:51:20.401778 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:20.401743 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 20:51:21.364793 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:21.364758 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-499187-55f6b9c44c-mmhjm"] Nov 03 20:51:21.368659 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:21.368644 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-499187-55f6b9c44c-mmhjm" Nov 03 20:51:21.426497 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:21.426458 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-499187-55f6b9c44c-mmhjm"] Nov 03 20:51:21.472970 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:21.472909 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-499187\"" Nov 03 20:51:21.473264 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:21.473045 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-499187-dockercfg-2zm8r\"" Nov 03 20:51:21.479787 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:21.479670 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad2a35f4-15e4-4130-ab26-9baba3182241-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-499187-55f6b9c44c-mmhjm\" (UID: \"ad2a35f4-15e4-4130-ab26-9baba3182241\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-499187-55f6b9c44c-mmhjm" Nov 03 20:51:21.480069 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:21.480046 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l4pg\" (UniqueName: \"kubernetes.io/projected/ad2a35f4-15e4-4130-ab26-9baba3182241-kube-api-access-5l4pg\") pod \"payload-pre-processing-e2e-worker-w0-499187-55f6b9c44c-mmhjm\" (UID: \"ad2a35f4-15e4-4130-ab26-9baba3182241\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-499187-55f6b9c44c-mmhjm" Nov 03 20:51:21.581116 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:21.581074 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l4pg\" (UniqueName: \"kubernetes.io/projected/ad2a35f4-15e4-4130-ab26-9baba3182241-kube-api-access-5l4pg\") pod \"payload-pre-processing-e2e-worker-w0-499187-55f6b9c44c-mmhjm\" (UID: \"ad2a35f4-15e4-4130-ab26-9baba3182241\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-499187-55f6b9c44c-mmhjm" Nov 03 20:51:21.581308 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:21.581203 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad2a35f4-15e4-4130-ab26-9baba3182241-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-499187-55f6b9c44c-mmhjm\" (UID: \"ad2a35f4-15e4-4130-ab26-9baba3182241\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-499187-55f6b9c44c-mmhjm" Nov 03 20:51:21.581914 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:21.581894 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad2a35f4-15e4-4130-ab26-9baba3182241-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-499187-55f6b9c44c-mmhjm\" (UID: \"ad2a35f4-15e4-4130-ab26-9baba3182241\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-499187-55f6b9c44c-mmhjm" Nov 03 20:51:21.642003 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:21.641949 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l4pg\" (UniqueName: \"kubernetes.io/projected/ad2a35f4-15e4-4130-ab26-9baba3182241-kube-api-access-5l4pg\") pod \"payload-pre-processing-e2e-worker-w0-499187-55f6b9c44c-mmhjm\" (UID: \"ad2a35f4-15e4-4130-ab26-9baba3182241\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-499187-55f6b9c44c-mmhjm" Nov 03 20:51:21.684052 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:21.683424 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-499187-55f6b9c44c-mmhjm" Nov 03 20:51:22.061463 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:22.061411 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-499187-55f6b9c44c-mmhjm"] Nov 03 20:51:22.063487 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:51:22.063462 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad2a35f4_15e4_4130_ab26_9baba3182241.slice/crio-88e3d4f4dfad34ed6806da70a6926c05ee1531e7a966fccecc116710208aadb7 WatchSource:0}: Error finding container 88e3d4f4dfad34ed6806da70a6926c05ee1531e7a966fccecc116710208aadb7: Status 404 returned error can't find the container with id 88e3d4f4dfad34ed6806da70a6926c05ee1531e7a966fccecc116710208aadb7 Nov 03 20:51:22.192762 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:22.192725 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-499187-55f6b9c44c-mmhjm" event={"ID":"ad2a35f4-15e4-4130-ab26-9baba3182241","Type":"ContainerStarted","Data":"aa5fc156bc5935b6331858f52c7586b02321bd59396a6012a3e88dbb666effdc"} Nov 03 20:51:22.192957 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:22.192887 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-499187-55f6b9c44c-mmhjm" Nov 03 20:51:22.192957 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:22.192935 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-499187-55f6b9c44c-mmhjm" event={"ID":"ad2a35f4-15e4-4130-ab26-9baba3182241","Type":"ContainerStarted","Data":"88e3d4f4dfad34ed6806da70a6926c05ee1531e7a966fccecc116710208aadb7"} Nov 03 20:51:22.311302 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:22.311200 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-499187-55f6b9c44c-mmhjm" podStartSLOduration=1.3111833210000001 podStartE2EDuration="1.311183321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:22.308643931 +0000 UTC m=+717.533963741" watchObservedRunningTime="2026-11-03 20:51:22.311183321 +0000 UTC m=+717.536503109" Nov 03 20:51:23.148580 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:23.148550 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ca4642-56449c8659-vtm4r" Nov 03 20:51:26.447770 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:26.447741 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-e22abf-c94f8f5d8-n4hfh"] Nov 03 20:51:26.451812 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:26.451791 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-e22abf-c94f8f5d8-n4hfh" Nov 03 20:51:26.530724 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:26.530688 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b5a3cd9-ead1-4bf0-ae49-f657449b964f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-e22abf-c94f8f5d8-n4hfh\" (UID: \"6b5a3cd9-ead1-4bf0-ae49-f657449b964f\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-e22abf-c94f8f5d8-n4hfh" Nov 03 20:51:26.530893 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:26.530872 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7cns\" (UniqueName: \"kubernetes.io/projected/6b5a3cd9-ead1-4bf0-ae49-f657449b964f-kube-api-access-m7cns\") pod \"payload-processing-e2e-worker-w5-e22abf-c94f8f5d8-n4hfh\" (UID: \"6b5a3cd9-ead1-4bf0-ae49-f657449b964f\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-e22abf-c94f8f5d8-n4hfh" Nov 03 20:51:26.555634 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:26.555592 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-e22abf-dockercfg-nxg9f\"" Nov 03 20:51:26.555830 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:26.555640 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-e22abf\"" Nov 03 20:51:26.558045 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:26.558019 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-e22abf-c94f8f5d8-n4hfh"] Nov 03 20:51:26.632356 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:26.632318 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b5a3cd9-ead1-4bf0-ae49-f657449b964f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-e22abf-c94f8f5d8-n4hfh\" (UID: \"6b5a3cd9-ead1-4bf0-ae49-f657449b964f\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-e22abf-c94f8f5d8-n4hfh" Nov 03 20:51:26.632570 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:26.632493 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7cns\" (UniqueName: \"kubernetes.io/projected/6b5a3cd9-ead1-4bf0-ae49-f657449b964f-kube-api-access-m7cns\") pod \"payload-processing-e2e-worker-w5-e22abf-c94f8f5d8-n4hfh\" (UID: \"6b5a3cd9-ead1-4bf0-ae49-f657449b964f\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-e22abf-c94f8f5d8-n4hfh" Nov 03 20:51:26.633063 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:26.633042 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b5a3cd9-ead1-4bf0-ae49-f657449b964f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-e22abf-c94f8f5d8-n4hfh\" (UID: \"6b5a3cd9-ead1-4bf0-ae49-f657449b964f\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-e22abf-c94f8f5d8-n4hfh" Nov 03 20:51:26.698244 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:26.698163 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7cns\" (UniqueName: \"kubernetes.io/projected/6b5a3cd9-ead1-4bf0-ae49-f657449b964f-kube-api-access-m7cns\") pod \"payload-processing-e2e-worker-w5-e22abf-c94f8f5d8-n4hfh\" (UID: \"6b5a3cd9-ead1-4bf0-ae49-f657449b964f\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-e22abf-c94f8f5d8-n4hfh" Nov 03 20:51:26.765539 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:26.765511 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-e22abf-c94f8f5d8-n4hfh" Nov 03 20:51:27.138862 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:27.138834 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-e22abf-c94f8f5d8-n4hfh"] Nov 03 20:51:27.141003 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:51:27.140952 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b5a3cd9_ead1_4bf0_ae49_f657449b964f.slice/crio-958081c54739e39b56a553576a034a0d370a4e3543a5a596ff42a147bfb7969f WatchSource:0}: Error finding container 958081c54739e39b56a553576a034a0d370a4e3543a5a596ff42a147bfb7969f: Status 404 returned error can't find the container with id 958081c54739e39b56a553576a034a0d370a4e3543a5a596ff42a147bfb7969f Nov 03 20:51:27.227482 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:27.227448 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-e22abf-c94f8f5d8-n4hfh" event={"ID":"6b5a3cd9-ead1-4bf0-ae49-f657449b964f","Type":"ContainerStarted","Data":"0a3a97cbc251b50547be8030373868870c1595488e0f14eeaab882fc5c1c9bc9"} Nov 03 20:51:27.227637 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:27.227514 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-e22abf-c94f8f5d8-n4hfh" event={"ID":"6b5a3cd9-ead1-4bf0-ae49-f657449b964f","Type":"ContainerStarted","Data":"958081c54739e39b56a553576a034a0d370a4e3543a5a596ff42a147bfb7969f"} Nov 03 20:51:27.227637 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:27.227570 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-e22abf-c94f8f5d8-n4hfh" Nov 03 20:51:27.355926 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:27.355857 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-e22abf-c94f8f5d8-n4hfh" podStartSLOduration=1.35584236 podStartE2EDuration="1.35584236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:27.352397069 +0000 UTC m=+722.577716887" watchObservedRunningTime="2026-11-03 20:51:27.35584236 +0000 UTC m=+722.581162145" Nov 03 20:51:28.174297 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:28.174267 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-12a423-559b96f96d-nmf2b" Nov 03 20:51:28.174738 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:28.174329 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-12a423-846d4b6867-grwcg" Nov 03 20:51:29.524767 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:29.524726 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 20:51:31.330980 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:31.330935 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-862138-6c8d9c5567-7nvj6"] Nov 03 20:51:31.335644 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:31.335618 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-862138-6c8d9c5567-7nvj6" Nov 03 20:51:31.435173 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:31.435134 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldqvn\" (UniqueName: \"kubernetes.io/projected/3e3a291b-d5be-4d9c-8b8a-0b97dba864e2-kube-api-access-ldqvn\") pod \"payload-pre-processing-e2e-worker-w2-862138-6c8d9c5567-7nvj6\" (UID: \"3e3a291b-d5be-4d9c-8b8a-0b97dba864e2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-862138-6c8d9c5567-7nvj6" Nov 03 20:51:31.435347 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:31.435215 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e3a291b-d5be-4d9c-8b8a-0b97dba864e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-862138-6c8d9c5567-7nvj6\" (UID: \"3e3a291b-d5be-4d9c-8b8a-0b97dba864e2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-862138-6c8d9c5567-7nvj6" Nov 03 20:51:31.455459 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:31.447325 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-862138\"" Nov 03 20:51:31.455459 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:31.447449 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-862138-dockercfg-thxff\"" Nov 03 20:51:31.478459 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:31.464525 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-862138-6c8d9c5567-7nvj6"] Nov 03 20:51:31.504514 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:31.501661 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-862138-65fb688c66-fx4kk"] Nov 03 20:51:31.540489 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:31.539323 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldqvn\" (UniqueName: \"kubernetes.io/projected/3e3a291b-d5be-4d9c-8b8a-0b97dba864e2-kube-api-access-ldqvn\") pod \"payload-pre-processing-e2e-worker-w2-862138-6c8d9c5567-7nvj6\" (UID: \"3e3a291b-d5be-4d9c-8b8a-0b97dba864e2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-862138-6c8d9c5567-7nvj6" Nov 03 20:51:31.540489 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:31.539416 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e3a291b-d5be-4d9c-8b8a-0b97dba864e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-862138-6c8d9c5567-7nvj6\" (UID: \"3e3a291b-d5be-4d9c-8b8a-0b97dba864e2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-862138-6c8d9c5567-7nvj6" Nov 03 20:51:31.541491 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:31.540768 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e3a291b-d5be-4d9c-8b8a-0b97dba864e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-862138-6c8d9c5567-7nvj6\" (UID: \"3e3a291b-d5be-4d9c-8b8a-0b97dba864e2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-862138-6c8d9c5567-7nvj6" Nov 03 20:51:31.543143 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:31.543124 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-862138-65fb688c66-fx4kk"] Nov 03 20:51:31.543374 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:31.543362 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-862138-65fb688c66-fx4kk" Nov 03 20:51:31.610862 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:31.610767 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldqvn\" (UniqueName: \"kubernetes.io/projected/3e3a291b-d5be-4d9c-8b8a-0b97dba864e2-kube-api-access-ldqvn\") pod \"payload-pre-processing-e2e-worker-w2-862138-6c8d9c5567-7nvj6\" (UID: \"3e3a291b-d5be-4d9c-8b8a-0b97dba864e2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-862138-6c8d9c5567-7nvj6" Nov 03 20:51:31.640160 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:31.640125 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c3bdfd9-c058-49d6-9179-4a53b523216f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-862138-65fb688c66-fx4kk\" (UID: \"6c3bdfd9-c058-49d6-9179-4a53b523216f\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-862138-65fb688c66-fx4kk" Nov 03 20:51:31.640327 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:31.640262 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4rc5\" (UniqueName: \"kubernetes.io/projected/6c3bdfd9-c058-49d6-9179-4a53b523216f-kube-api-access-z4rc5\") pod \"payload-processing-e2e-worker-w2-862138-65fb688c66-fx4kk\" (UID: \"6c3bdfd9-c058-49d6-9179-4a53b523216f\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-862138-65fb688c66-fx4kk" Nov 03 20:51:31.676461 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:31.675649 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-862138-6c8d9c5567-7nvj6" Nov 03 20:51:31.741091 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:31.741021 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4rc5\" (UniqueName: \"kubernetes.io/projected/6c3bdfd9-c058-49d6-9179-4a53b523216f-kube-api-access-z4rc5\") pod \"payload-processing-e2e-worker-w2-862138-65fb688c66-fx4kk\" (UID: \"6c3bdfd9-c058-49d6-9179-4a53b523216f\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-862138-65fb688c66-fx4kk" Nov 03 20:51:31.741240 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:31.741174 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c3bdfd9-c058-49d6-9179-4a53b523216f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-862138-65fb688c66-fx4kk\" (UID: \"6c3bdfd9-c058-49d6-9179-4a53b523216f\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-862138-65fb688c66-fx4kk" Nov 03 20:51:31.742429 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:31.742407 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c3bdfd9-c058-49d6-9179-4a53b523216f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-862138-65fb688c66-fx4kk\" (UID: \"6c3bdfd9-c058-49d6-9179-4a53b523216f\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-862138-65fb688c66-fx4kk" Nov 03 20:51:31.813183 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:31.813115 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4rc5\" (UniqueName: \"kubernetes.io/projected/6c3bdfd9-c058-49d6-9179-4a53b523216f-kube-api-access-z4rc5\") pod \"payload-processing-e2e-worker-w2-862138-65fb688c66-fx4kk\" (UID: \"6c3bdfd9-c058-49d6-9179-4a53b523216f\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-862138-65fb688c66-fx4kk" Nov 03 20:51:31.912918 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:31.912882 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-862138-65fb688c66-fx4kk" Nov 03 20:51:32.183529 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:32.178932 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-862138-6c8d9c5567-7nvj6"] Nov 03 20:51:32.207415 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:51:32.207375 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e3a291b_d5be_4d9c_8b8a_0b97dba864e2.slice/crio-73704d0429ed3c325c85b49896bbaa436f0f1742d97594c04544b19d6a420269 WatchSource:0}: Error finding container 73704d0429ed3c325c85b49896bbaa436f0f1742d97594c04544b19d6a420269: Status 404 returned error can't find the container with id 73704d0429ed3c325c85b49896bbaa436f0f1742d97594c04544b19d6a420269 Nov 03 20:51:32.334049 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:32.334018 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-862138-6c8d9c5567-7nvj6" event={"ID":"3e3a291b-d5be-4d9c-8b8a-0b97dba864e2","Type":"ContainerStarted","Data":"fd4421526c4ecde741d5b2716623486237c2c7688cb46b1c2f684a68ed674acd"} Nov 03 20:51:32.334461 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:32.334068 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-862138-6c8d9c5567-7nvj6" event={"ID":"3e3a291b-d5be-4d9c-8b8a-0b97dba864e2","Type":"ContainerStarted","Data":"73704d0429ed3c325c85b49896bbaa436f0f1742d97594c04544b19d6a420269"} Nov 03 20:51:32.334461 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:32.334114 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-862138-6c8d9c5567-7nvj6" Nov 03 20:51:32.342890 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:51:32.342860 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c3bdfd9_c058_49d6_9179_4a53b523216f.slice/crio-7f7dd5f3705a424f45e9ada4c5279052c881f0fcb58ea3b202d5042e7fcf341f WatchSource:0}: Error finding container 7f7dd5f3705a424f45e9ada4c5279052c881f0fcb58ea3b202d5042e7fcf341f: Status 404 returned error can't find the container with id 7f7dd5f3705a424f45e9ada4c5279052c881f0fcb58ea3b202d5042e7fcf341f Nov 03 20:51:32.343264 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:32.343247 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-862138-65fb688c66-fx4kk"] Nov 03 20:51:32.505462 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:32.497534 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-862138-6c8d9c5567-7nvj6" podStartSLOduration=1.497510997 podStartE2EDuration="1.497510997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:32.485854814 +0000 UTC m=+727.711174626" watchObservedRunningTime="2026-11-03 20:51:32.497510997 +0000 UTC m=+727.722830786" Nov 03 20:51:33.204831 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:33.204801 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-499187-55f6b9c44c-mmhjm" Nov 03 20:51:33.343169 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:33.343127 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-862138-65fb688c66-fx4kk" event={"ID":"6c3bdfd9-c058-49d6-9179-4a53b523216f","Type":"ContainerStarted","Data":"1b30d77ad6e760a8dec5065402b6f262ec0231531ac296a326ee1ad0abfa0e98"} Nov 03 20:51:33.343861 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:33.343817 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-862138-65fb688c66-fx4kk" event={"ID":"6c3bdfd9-c058-49d6-9179-4a53b523216f","Type":"ContainerStarted","Data":"7f7dd5f3705a424f45e9ada4c5279052c881f0fcb58ea3b202d5042e7fcf341f"} Nov 03 20:51:33.344093 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:33.344068 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-862138-65fb688c66-fx4kk" Nov 03 20:51:33.500520 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:33.492996 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-862138-65fb688c66-fx4kk" podStartSLOduration=2.492960064 podStartE2EDuration="2.492960064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:33.474858 +0000 UTC m=+728.700177818" watchObservedRunningTime="2026-11-03 20:51:33.492960064 +0000 UTC m=+728.718279855" Nov 03 20:51:36.538782 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:36.538741 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-d01cef-848dd5d487-swhfc"] Nov 03 20:51:36.543231 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:36.543202 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d01cef-848dd5d487-swhfc" Nov 03 20:51:36.646472 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:36.646396 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-d01cef\"" Nov 03 20:51:36.646472 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:36.646425 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-d01cef-dockercfg-6l5tr\"" Nov 03 20:51:36.650154 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:36.650116 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-d01cef-848dd5d487-swhfc"] Nov 03 20:51:36.653256 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:36.653219 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgmrn\" (UniqueName: \"kubernetes.io/projected/aab129fc-dcb8-4cd0-beaa-49f36dfb3f38-kube-api-access-mgmrn\") pod \"payload-pre-processing-e2e-worker-w1-d01cef-848dd5d487-swhfc\" (UID: \"aab129fc-dcb8-4cd0-beaa-49f36dfb3f38\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d01cef-848dd5d487-swhfc" Nov 03 20:51:36.653424 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:36.653358 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aab129fc-dcb8-4cd0-beaa-49f36dfb3f38-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-d01cef-848dd5d487-swhfc\" (UID: \"aab129fc-dcb8-4cd0-beaa-49f36dfb3f38\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d01cef-848dd5d487-swhfc" Nov 03 20:51:36.717457 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:36.712079 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-d01cef-996dcbc66-xfqll"] Nov 03 20:51:36.718458 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:36.717855 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-d01cef-996dcbc66-xfqll" Nov 03 20:51:36.761465 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:36.740860 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-d01cef-996dcbc66-xfqll"] Nov 03 20:51:36.761465 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:36.755604 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aab129fc-dcb8-4cd0-beaa-49f36dfb3f38-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-d01cef-848dd5d487-swhfc\" (UID: \"aab129fc-dcb8-4cd0-beaa-49f36dfb3f38\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d01cef-848dd5d487-swhfc" Nov 03 20:51:36.761465 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:36.755800 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gql4p\" (UniqueName: \"kubernetes.io/projected/ffdb1619-de55-4707-8dbf-b40be40c608e-kube-api-access-gql4p\") pod \"payload-processing-e2e-worker-w1-d01cef-996dcbc66-xfqll\" (UID: \"ffdb1619-de55-4707-8dbf-b40be40c608e\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-d01cef-996dcbc66-xfqll" Nov 03 20:51:36.761465 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:36.755990 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgmrn\" (UniqueName: \"kubernetes.io/projected/aab129fc-dcb8-4cd0-beaa-49f36dfb3f38-kube-api-access-mgmrn\") pod \"payload-pre-processing-e2e-worker-w1-d01cef-848dd5d487-swhfc\" (UID: \"aab129fc-dcb8-4cd0-beaa-49f36dfb3f38\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d01cef-848dd5d487-swhfc" Nov 03 20:51:36.761465 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:36.756054 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffdb1619-de55-4707-8dbf-b40be40c608e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-d01cef-996dcbc66-xfqll\" (UID: \"ffdb1619-de55-4707-8dbf-b40be40c608e\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-d01cef-996dcbc66-xfqll" Nov 03 20:51:36.761465 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:36.758408 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aab129fc-dcb8-4cd0-beaa-49f36dfb3f38-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-d01cef-848dd5d487-swhfc\" (UID: \"aab129fc-dcb8-4cd0-beaa-49f36dfb3f38\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d01cef-848dd5d487-swhfc" Nov 03 20:51:36.821955 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:36.821880 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgmrn\" (UniqueName: \"kubernetes.io/projected/aab129fc-dcb8-4cd0-beaa-49f36dfb3f38-kube-api-access-mgmrn\") pod \"payload-pre-processing-e2e-worker-w1-d01cef-848dd5d487-swhfc\" (UID: \"aab129fc-dcb8-4cd0-beaa-49f36dfb3f38\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d01cef-848dd5d487-swhfc" Nov 03 20:51:36.857237 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:36.857190 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gql4p\" (UniqueName: \"kubernetes.io/projected/ffdb1619-de55-4707-8dbf-b40be40c608e-kube-api-access-gql4p\") pod \"payload-processing-e2e-worker-w1-d01cef-996dcbc66-xfqll\" (UID: \"ffdb1619-de55-4707-8dbf-b40be40c608e\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-d01cef-996dcbc66-xfqll" Nov 03 20:51:36.857474 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:36.857288 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffdb1619-de55-4707-8dbf-b40be40c608e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-d01cef-996dcbc66-xfqll\" (UID: \"ffdb1619-de55-4707-8dbf-b40be40c608e\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-d01cef-996dcbc66-xfqll" Nov 03 20:51:36.857474 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:36.857378 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d01cef-848dd5d487-swhfc" Nov 03 20:51:36.858126 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:36.858105 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffdb1619-de55-4707-8dbf-b40be40c608e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-d01cef-996dcbc66-xfqll\" (UID: \"ffdb1619-de55-4707-8dbf-b40be40c608e\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-d01cef-996dcbc66-xfqll" Nov 03 20:51:36.920757 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:36.920721 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gql4p\" (UniqueName: \"kubernetes.io/projected/ffdb1619-de55-4707-8dbf-b40be40c608e-kube-api-access-gql4p\") pod \"payload-processing-e2e-worker-w1-d01cef-996dcbc66-xfqll\" (UID: \"ffdb1619-de55-4707-8dbf-b40be40c608e\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-d01cef-996dcbc66-xfqll" Nov 03 20:51:37.052751 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:37.052712 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-d01cef-996dcbc66-xfqll" Nov 03 20:51:37.260875 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:37.260843 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-d01cef-848dd5d487-swhfc"] Nov 03 20:51:37.262527 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:51:37.262491 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaab129fc_dcb8_4cd0_beaa_49f36dfb3f38.slice/crio-d3ca8133974550493164dd0566b879f55d730d4a44aefa67012f9b68d7d0a632 WatchSource:0}: Error finding container d3ca8133974550493164dd0566b879f55d730d4a44aefa67012f9b68d7d0a632: Status 404 returned error can't find the container with id d3ca8133974550493164dd0566b879f55d730d4a44aefa67012f9b68d7d0a632 Nov 03 20:51:37.376699 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:37.376603 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d01cef-848dd5d487-swhfc" event={"ID":"aab129fc-dcb8-4cd0-beaa-49f36dfb3f38","Type":"ContainerStarted","Data":"ae84a85c3c2b8975a14902e1a23cbc07af7b701ed99cf7971ee6e5a902538628"} Nov 03 20:51:37.376699 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:37.376673 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d01cef-848dd5d487-swhfc" event={"ID":"aab129fc-dcb8-4cd0-beaa-49f36dfb3f38","Type":"ContainerStarted","Data":"d3ca8133974550493164dd0566b879f55d730d4a44aefa67012f9b68d7d0a632"} Nov 03 20:51:37.376905 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:37.376832 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d01cef-848dd5d487-swhfc" Nov 03 20:51:37.438350 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:37.438311 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-d01cef-996dcbc66-xfqll"] Nov 03 20:51:37.441587 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:51:37.441554 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffdb1619_de55_4707_8dbf_b40be40c608e.slice/crio-484fbbf7bbc427b9847e6defcc5e21094822e867bcdb116d53130933058f530f WatchSource:0}: Error finding container 484fbbf7bbc427b9847e6defcc5e21094822e867bcdb116d53130933058f530f: Status 404 returned error can't find the container with id 484fbbf7bbc427b9847e6defcc5e21094822e867bcdb116d53130933058f530f Nov 03 20:51:37.497701 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:37.497647 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d01cef-848dd5d487-swhfc" podStartSLOduration=1.497627032 podStartE2EDuration="1.497627032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:37.494293712 +0000 UTC m=+732.719613525" watchObservedRunningTime="2026-11-03 20:51:37.497627032 +0000 UTC m=+732.722946819" Nov 03 20:51:38.236568 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:38.236538 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-e22abf-c94f8f5d8-n4hfh" Nov 03 20:51:38.386255 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:38.386218 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-d01cef-996dcbc66-xfqll" event={"ID":"ffdb1619-de55-4707-8dbf-b40be40c608e","Type":"ContainerStarted","Data":"914a4233ec2fe008a13bfdb1e3698a77b22c3f8e2869726776136fe677111ed5"} Nov 03 20:51:38.386463 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:38.386287 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-d01cef-996dcbc66-xfqll" event={"ID":"ffdb1619-de55-4707-8dbf-b40be40c608e","Type":"ContainerStarted","Data":"484fbbf7bbc427b9847e6defcc5e21094822e867bcdb116d53130933058f530f"} Nov 03 20:51:38.386463 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:38.386319 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-d01cef-996dcbc66-xfqll" Nov 03 20:51:38.507303 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:38.507190 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-d01cef-996dcbc66-xfqll" podStartSLOduration=2.507172752 podStartE2EDuration="2.507172752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:38.504914308 +0000 UTC m=+733.730234134" watchObservedRunningTime="2026-11-03 20:51:38.507172752 +0000 UTC m=+733.732492542" Nov 03 20:51:39.392379 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:39.392346 4305 generic.go:356] "Generic (PLEG): container finished" podID="58829723-0628-457e-80b8-f81d15ecfe2e" containerID="10d10b551cde5c66d72ab000240f018c97e9ffa11aafc083cfab2f0a0f710fe3" exitCode=0 Nov 03 20:51:39.392897 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:39.392422 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28" event={"ID":"58829723-0628-457e-80b8-f81d15ecfe2e","Type":"ContainerDied","Data":"10d10b551cde5c66d72ab000240f018c97e9ffa11aafc083cfab2f0a0f710fe3"} Nov 03 20:51:39.392897 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:39.392848 4305 scope.go:122] "RemoveContainer" containerID="10d10b551cde5c66d72ab000240f018c97e9ffa11aafc083cfab2f0a0f710fe3" Nov 03 20:51:39.394069 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:39.394048 4305 generic.go:356] "Generic (PLEG): container finished" podID="e4b14774-55f2-4fad-88ad-74d08bf328f7" containerID="b25dde8b6a1675398d99debfe7fc44da9de0edb7814bcfe8cb24683fafbc35d3" exitCode=0 Nov 03 20:51:39.394156 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:39.394114 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p" event={"ID":"e4b14774-55f2-4fad-88ad-74d08bf328f7","Type":"ContainerDied","Data":"b25dde8b6a1675398d99debfe7fc44da9de0edb7814bcfe8cb24683fafbc35d3"} Nov 03 20:51:39.394240 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:39.394216 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:51:39.394295 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:39.394283 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:51:39.394328 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:39.394317 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:51:39.394541 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:39.394527 4305 scope.go:122] "RemoveContainer" containerID="b25dde8b6a1675398d99debfe7fc44da9de0edb7814bcfe8cb24683fafbc35d3" Nov 03 20:51:39.395642 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:39.395619 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:51:39.395698 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:39.395678 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:51:39.395731 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:39.395710 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:51:40.321183 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:40.321149 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 20:51:40.410499 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:40.410470 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28" event={"ID":"58829723-0628-457e-80b8-f81d15ecfe2e","Type":"ContainerStarted","Data":"bf7eed539704b28314d8666f8366e928a9a1acdf0bd6773de39b3c6c370f01a9"} Nov 03 20:51:40.412936 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:40.412908 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p" event={"ID":"e4b14774-55f2-4fad-88ad-74d08bf328f7","Type":"ContainerStarted","Data":"2abb835f949043e4a77b448f621626fa8dd5fca6fc7a6fd8ddc5da199fc88179"} Nov 03 20:51:40.974461 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:40.972092 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28" Nov 03 20:51:40.974461 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:40.972219 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 03 20:51:40.974461 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:40.972275 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28" podUID="58829723-0628-457e-80b8-f81d15ecfe2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 03 20:51:41.275976 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:41.275868 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p" Nov 03 20:51:41.276327 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:41.276306 4305 patch_prober.go:29] interesting pod/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 03 20:51:41.276514 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:41.276491 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p" podUID="e4b14774-55f2-4fad-88ad-74d08bf328f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 03 20:51:41.975961 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:41.975919 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 03 20:51:41.976697 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:41.976661 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28" podUID="58829723-0628-457e-80b8-f81d15ecfe2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 03 20:51:42.280517 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:42.279557 4305 patch_prober.go:29] interesting pod/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 03 20:51:42.280517 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:42.279622 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p" podUID="e4b14774-55f2-4fad-88ad-74d08bf328f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 03 20:51:42.973459 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:42.972820 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 03 20:51:42.973459 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:42.972923 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28" podUID="58829723-0628-457e-80b8-f81d15ecfe2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 03 20:51:43.284525 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:43.275891 4305 patch_prober.go:29] interesting pod/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 03 20:51:43.284525 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:43.275958 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p" podUID="e4b14774-55f2-4fad-88ad-74d08bf328f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 03 20:51:43.346283 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:43.346256 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-862138-6c8d9c5567-7nvj6" Nov 03 20:51:43.974570 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:43.974535 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 03 20:51:43.974860 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:43.974831 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28" podUID="58829723-0628-457e-80b8-f81d15ecfe2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 03 20:51:44.276088 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:44.275982 4305 patch_prober.go:29] interesting pod/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 03 20:51:44.276367 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:44.276341 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p" podUID="e4b14774-55f2-4fad-88ad-74d08bf328f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 03 20:51:44.972153 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:44.972113 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 03 20:51:44.972759 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:44.972729 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28" podUID="58829723-0628-457e-80b8-f81d15ecfe2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 03 20:51:45.279513 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:45.279078 4305 patch_prober.go:29] interesting pod/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 03 20:51:45.279513 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:45.279139 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p" podUID="e4b14774-55f2-4fad-88ad-74d08bf328f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 03 20:51:45.391279 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:45.391240 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-862138-65fb688c66-fx4kk" Nov 03 20:51:45.976370 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:45.976342 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28" Nov 03 20:51:45.976816 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:45.976668 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28" Nov 03 20:51:45.977290 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:45.977270 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28" Nov 03 20:51:46.279804 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:46.279718 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p" Nov 03 20:51:46.280198 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:46.280175 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p" Nov 03 20:51:46.280924 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:46.280904 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p" Nov 03 20:51:46.321766 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:51:46.321730 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 20:51:48.390731 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:48.390703 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d01cef-848dd5d487-swhfc" Nov 03 20:51:49.397889 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:49.397861 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-d01cef-996dcbc66-xfqll" Nov 03 20:51:50.394993 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:51:50.394953 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 20:52:00.550151 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:52:00.550047 4305 generic.go:356] "Generic (PLEG): container finished" podID="365e3804-37c5-4398-95c0-a1602237453e" containerID="8c843261fb8ad8e8ef30b9bc672917ec5843dfc9995c59389d3b8fe006d9a0f2" exitCode=0 Nov 03 20:52:00.550151 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:52:00.550124 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft" event={"ID":"365e3804-37c5-4398-95c0-a1602237453e","Type":"ContainerDied","Data":"8c843261fb8ad8e8ef30b9bc672917ec5843dfc9995c59389d3b8fe006d9a0f2"} Nov 03 20:52:00.550648 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:52:00.550566 4305 scope.go:122] "RemoveContainer" containerID="8c843261fb8ad8e8ef30b9bc672917ec5843dfc9995c59389d3b8fe006d9a0f2" Nov 03 20:52:00.551929 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:52:00.551898 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:52:00.552026 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:52:00.551967 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:52:00.552026 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:52:00.552005 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:52:01.582536 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:52:01.582495 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft" event={"ID":"365e3804-37c5-4398-95c0-a1602237453e","Type":"ContainerStarted","Data":"097e2b681c7777e9b661d86817933bb1d1b6cbd767c560ff92ea59a4807a981a"} Nov 03 20:52:02.333345 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:52:02.333266 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft" Nov 03 20:52:02.334711 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:52:02.334662 4305 patch_prober.go:29] interesting pod/e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 03 20:52:02.334900 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:52:02.334729 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft" podUID="365e3804-37c5-4398-95c0-a1602237453e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 03 20:52:03.088349 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:52:03.088320 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft" Nov 03 20:52:03.332809 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:52:03.332773 4305 patch_prober.go:29] interesting pod/e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 03 20:52:03.332993 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:52:03.332834 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft" podUID="365e3804-37c5-4398-95c0-a1602237453e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 03 20:52:04.333240 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:52:04.333199 4305 patch_prober.go:29] interesting pod/e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 03 20:52:04.333724 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:52:04.333259 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft" podUID="365e3804-37c5-4398-95c0-a1602237453e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 03 20:52:04.901295 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:52:04.901252 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 20:52:05.333128 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:52:05.333037 4305 patch_prober.go:29] interesting pod/e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 03 20:52:05.333449 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:52:05.333398 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft" podUID="365e3804-37c5-4398-95c0-a1602237453e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 03 20:52:06.333234 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:52:06.333150 4305 patch_prober.go:29] interesting pod/e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 03 20:52:06.333234 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:52:06.333220 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft" podUID="365e3804-37c5-4398-95c0-a1602237453e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 03 20:52:07.336960 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:52:07.336926 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft" Nov 03 20:52:07.338236 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:52:07.338216 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft" Nov 03 20:52:16.322787 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:52:16.322744 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 20:52:16.405711 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:52:16.405671 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 20:52:38.539701 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:52:38.539666 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 20:52:46.321095 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:52:46.321058 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 20:52:47.602397 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:52:47.602367 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8"] Nov 03 20:52:47.606364 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:52:47.606344 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" Nov 03 20:52:47.689269 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:52:47.689231 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8"] Nov 03 20:52:47.708313 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:52:47.708283 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-499187\"/\"openshift-service-ca.crt\"" Nov 03 20:52:47.708530 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:52:47.708388 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-499187\"/\"simulator-w0-499187-kserve-self-signed-certs\"" Nov 03 20:52:47.708530 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:52:47.708494 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-499187\"/\"default-dockercfg-6gx7b\"" Nov 03 20:52:47.711179 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:52:47.711163 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-499187\"/\"kube-root-ca.crt\"" Nov 03 20:52:47.715689 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:52:47.715670 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/793b8ca3-3a2e-49a1-9be4-4971ba59df09-dshm\") pod \"simulator-w0-499187-kserve-677fb598c-cxgd8\" (UID: \"793b8ca3-3a2e-49a1-9be4-4971ba59df09\") " pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" Nov 03 20:52:47.715763 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:52:47.715707 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/793b8ca3-3a2e-49a1-9be4-4971ba59df09-tmp-dir\") pod \"simulator-w0-499187-kserve-677fb598c-cxgd8\" (UID: \"793b8ca3-3a2e-49a1-9be4-4971ba59df09\") " pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" Nov 03 20:52:47.715883 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:52:47.715864 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkb5c\" (UniqueName: \"kubernetes.io/projected/793b8ca3-3a2e-49a1-9be4-4971ba59df09-kube-api-access-vkb5c\") pod \"simulator-w0-499187-kserve-677fb598c-cxgd8\" (UID: \"793b8ca3-3a2e-49a1-9be4-4971ba59df09\") " pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" Nov 03 20:52:47.715920 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:52:47.715900 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/793b8ca3-3a2e-49a1-9be4-4971ba59df09-tls-certs\") pod \"simulator-w0-499187-kserve-677fb598c-cxgd8\" (UID: \"793b8ca3-3a2e-49a1-9be4-4971ba59df09\") " pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" Nov 03 20:52:47.715956 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:52:47.715928 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/793b8ca3-3a2e-49a1-9be4-4971ba59df09-model-cache\") pod \"simulator-w0-499187-kserve-677fb598c-cxgd8\" (UID: \"793b8ca3-3a2e-49a1-9be4-4971ba59df09\") " pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" Nov 03 20:52:47.716003 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:52:47.715987 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/793b8ca3-3a2e-49a1-9be4-4971ba59df09-home\") pod \"simulator-w0-499187-kserve-677fb598c-cxgd8\" (UID: \"793b8ca3-3a2e-49a1-9be4-4971ba59df09\") " pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" Nov 03 20:52:47.816744 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:52:47.816711 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/793b8ca3-3a2e-49a1-9be4-4971ba59df09-tmp-dir\") pod \"simulator-w0-499187-kserve-677fb598c-cxgd8\" (UID: \"793b8ca3-3a2e-49a1-9be4-4971ba59df09\") " pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" Nov 03 20:52:47.816928 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:52:47.816814 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkb5c\" (UniqueName: \"kubernetes.io/projected/793b8ca3-3a2e-49a1-9be4-4971ba59df09-kube-api-access-vkb5c\") pod \"simulator-w0-499187-kserve-677fb598c-cxgd8\" (UID: \"793b8ca3-3a2e-49a1-9be4-4971ba59df09\") " pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" Nov 03 20:52:47.816928 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:52:47.816838 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/793b8ca3-3a2e-49a1-9be4-4971ba59df09-tls-certs\") pod \"simulator-w0-499187-kserve-677fb598c-cxgd8\" (UID: \"793b8ca3-3a2e-49a1-9be4-4971ba59df09\") " pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" Nov 03 20:52:47.816928 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:52:47.816866 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/793b8ca3-3a2e-49a1-9be4-4971ba59df09-model-cache\") pod \"simulator-w0-499187-kserve-677fb598c-cxgd8\" (UID: \"793b8ca3-3a2e-49a1-9be4-4971ba59df09\") " pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" Nov 03 20:52:47.816928 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:52:47.816886 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/793b8ca3-3a2e-49a1-9be4-4971ba59df09-home\") pod \"simulator-w0-499187-kserve-677fb598c-cxgd8\" (UID: \"793b8ca3-3a2e-49a1-9be4-4971ba59df09\") " pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" Nov 03 20:52:47.817142 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:52:47.816933 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/793b8ca3-3a2e-49a1-9be4-4971ba59df09-dshm\") pod \"simulator-w0-499187-kserve-677fb598c-cxgd8\" (UID: \"793b8ca3-3a2e-49a1-9be4-4971ba59df09\") " pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" Nov 03 20:52:47.817192 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:52:47.817164 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/793b8ca3-3a2e-49a1-9be4-4971ba59df09-tmp-dir\") pod \"simulator-w0-499187-kserve-677fb598c-cxgd8\" (UID: \"793b8ca3-3a2e-49a1-9be4-4971ba59df09\") " pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" Nov 03 20:52:47.817296 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:52:47.817272 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/793b8ca3-3a2e-49a1-9be4-4971ba59df09-home\") pod \"simulator-w0-499187-kserve-677fb598c-cxgd8\" (UID: \"793b8ca3-3a2e-49a1-9be4-4971ba59df09\") " pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" Nov 03 20:52:47.817380 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:52:47.817358 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/793b8ca3-3a2e-49a1-9be4-4971ba59df09-model-cache\") pod \"simulator-w0-499187-kserve-677fb598c-cxgd8\" (UID: \"793b8ca3-3a2e-49a1-9be4-4971ba59df09\") " pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" Nov 03 20:52:47.819343 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:52:47.819318 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/793b8ca3-3a2e-49a1-9be4-4971ba59df09-dshm\") pod \"simulator-w0-499187-kserve-677fb598c-cxgd8\" (UID: \"793b8ca3-3a2e-49a1-9be4-4971ba59df09\") " pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" Nov 03 20:52:47.819729 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:52:47.819709 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/793b8ca3-3a2e-49a1-9be4-4971ba59df09-tls-certs\") pod \"simulator-w0-499187-kserve-677fb598c-cxgd8\" (UID: \"793b8ca3-3a2e-49a1-9be4-4971ba59df09\") " pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" Nov 03 20:52:47.885702 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:52:47.885634 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkb5c\" (UniqueName: \"kubernetes.io/projected/793b8ca3-3a2e-49a1-9be4-4971ba59df09-kube-api-access-vkb5c\") pod \"simulator-w0-499187-kserve-677fb598c-cxgd8\" (UID: \"793b8ca3-3a2e-49a1-9be4-4971ba59df09\") " pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" Nov 03 20:52:47.917662 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:52:47.917624 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" Nov 03 20:52:48.318072 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:52:48.318050 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8"] Nov 03 20:52:48.320164 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:52:48.320133 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod793b8ca3_3a2e_49a1_9be4_4971ba59df09.slice/crio-56998c882f0462d0edc1b03d6b6819e0ccde86c0e76a4b932e98e18c1f54e96f WatchSource:0}: Error finding container 56998c882f0462d0edc1b03d6b6819e0ccde86c0e76a4b932e98e18c1f54e96f: Status 404 returned error can't find the container with id 56998c882f0462d0edc1b03d6b6819e0ccde86c0e76a4b932e98e18c1f54e96f Nov 03 20:52:48.859074 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:52:48.859043 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" event={"ID":"793b8ca3-3a2e-49a1-9be4-4971ba59df09","Type":"ContainerStarted","Data":"2b093b23e776f3f66dccf41616165d5e03634758a3a061ec4f6eeb0dae0ff152"} Nov 03 20:52:48.859074 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:52:48.859080 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" event={"ID":"793b8ca3-3a2e-49a1-9be4-4971ba59df09","Type":"ContainerStarted","Data":"56998c882f0462d0edc1b03d6b6819e0ccde86c0e76a4b932e98e18c1f54e96f"} Nov 03 20:52:48.918402 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:52:48.918368 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" Nov 03 20:52:48.987317 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:52:48.987264 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" podStartSLOduration=1.987249421 podStartE2EDuration="1.987249421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:48.986131876 +0000 UTC m=+804.211451668" watchObservedRunningTime="2026-11-03 20:52:48.987249421 +0000 UTC m=+804.212569212" Nov 03 20:52:57.918100 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:52:57.918059 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" Nov 03 20:52:57.931148 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:52:57.931119 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" Nov 03 20:52:58.925192 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:52:58.925164 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" Nov 03 20:53:04.548737 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:04.548706 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 20:53:05.845326 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:05.845291 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc"] Nov 03 20:53:05.849169 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:05.849143 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" Nov 03 20:53:05.921218 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:05.921184 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc"] Nov 03 20:53:05.951104 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:05.951073 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-ca4642\"/\"openshift-service-ca.crt\"" Nov 03 20:53:05.951507 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:05.951481 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-ca4642\"/\"kube-root-ca.crt\"" Nov 03 20:53:05.951959 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:05.951939 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ca4642\"/\"simulator-w4-ca4642-kserve-self-signed-certs\"" Nov 03 20:53:05.954223 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:05.954191 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ca4642\"/\"default-dockercfg-nq5lb\"" Nov 03 20:53:06.023970 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:06.023929 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lxcn\" (UniqueName: \"kubernetes.io/projected/80de7944-70f6-49e1-a0e8-4b82d5caad0d-kube-api-access-5lxcn\") pod \"simulator-w4-ca4642-kserve-6f45856c59-2xbqc\" (UID: \"80de7944-70f6-49e1-a0e8-4b82d5caad0d\") " pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" Nov 03 20:53:06.024163 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:06.024054 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80de7944-70f6-49e1-a0e8-4b82d5caad0d-home\") pod \"simulator-w4-ca4642-kserve-6f45856c59-2xbqc\" (UID: \"80de7944-70f6-49e1-a0e8-4b82d5caad0d\") " pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" Nov 03 20:53:06.024163 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:06.024135 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80de7944-70f6-49e1-a0e8-4b82d5caad0d-dshm\") pod \"simulator-w4-ca4642-kserve-6f45856c59-2xbqc\" (UID: \"80de7944-70f6-49e1-a0e8-4b82d5caad0d\") " pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" Nov 03 20:53:06.024278 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:06.024169 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80de7944-70f6-49e1-a0e8-4b82d5caad0d-tls-certs\") pod \"simulator-w4-ca4642-kserve-6f45856c59-2xbqc\" (UID: \"80de7944-70f6-49e1-a0e8-4b82d5caad0d\") " pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" Nov 03 20:53:06.024402 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:06.024380 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80de7944-70f6-49e1-a0e8-4b82d5caad0d-tmp-dir\") pod \"simulator-w4-ca4642-kserve-6f45856c59-2xbqc\" (UID: \"80de7944-70f6-49e1-a0e8-4b82d5caad0d\") " pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" Nov 03 20:53:06.024608 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:06.024591 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80de7944-70f6-49e1-a0e8-4b82d5caad0d-model-cache\") pod \"simulator-w4-ca4642-kserve-6f45856c59-2xbqc\" (UID: \"80de7944-70f6-49e1-a0e8-4b82d5caad0d\") " pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" Nov 03 20:53:06.125286 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:06.125194 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80de7944-70f6-49e1-a0e8-4b82d5caad0d-model-cache\") pod \"simulator-w4-ca4642-kserve-6f45856c59-2xbqc\" (UID: \"80de7944-70f6-49e1-a0e8-4b82d5caad0d\") " pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" Nov 03 20:53:06.125286 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:06.125243 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lxcn\" (UniqueName: \"kubernetes.io/projected/80de7944-70f6-49e1-a0e8-4b82d5caad0d-kube-api-access-5lxcn\") pod \"simulator-w4-ca4642-kserve-6f45856c59-2xbqc\" (UID: \"80de7944-70f6-49e1-a0e8-4b82d5caad0d\") " pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" Nov 03 20:53:06.125570 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:06.125299 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80de7944-70f6-49e1-a0e8-4b82d5caad0d-home\") pod \"simulator-w4-ca4642-kserve-6f45856c59-2xbqc\" (UID: \"80de7944-70f6-49e1-a0e8-4b82d5caad0d\") " pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" Nov 03 20:53:06.125570 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:06.125327 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80de7944-70f6-49e1-a0e8-4b82d5caad0d-dshm\") pod \"simulator-w4-ca4642-kserve-6f45856c59-2xbqc\" (UID: \"80de7944-70f6-49e1-a0e8-4b82d5caad0d\") " pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" Nov 03 20:53:06.125570 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:06.125356 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80de7944-70f6-49e1-a0e8-4b82d5caad0d-tls-certs\") pod \"simulator-w4-ca4642-kserve-6f45856c59-2xbqc\" (UID: \"80de7944-70f6-49e1-a0e8-4b82d5caad0d\") " pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" Nov 03 20:53:06.125570 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:06.125411 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80de7944-70f6-49e1-a0e8-4b82d5caad0d-tmp-dir\") pod \"simulator-w4-ca4642-kserve-6f45856c59-2xbqc\" (UID: \"80de7944-70f6-49e1-a0e8-4b82d5caad0d\") " pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" Nov 03 20:53:06.125789 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:06.125706 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80de7944-70f6-49e1-a0e8-4b82d5caad0d-model-cache\") pod \"simulator-w4-ca4642-kserve-6f45856c59-2xbqc\" (UID: \"80de7944-70f6-49e1-a0e8-4b82d5caad0d\") " pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" Nov 03 20:53:06.125789 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:06.125772 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80de7944-70f6-49e1-a0e8-4b82d5caad0d-home\") pod \"simulator-w4-ca4642-kserve-6f45856c59-2xbqc\" (UID: \"80de7944-70f6-49e1-a0e8-4b82d5caad0d\") " pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" Nov 03 20:53:06.125882 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:06.125844 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80de7944-70f6-49e1-a0e8-4b82d5caad0d-tmp-dir\") pod \"simulator-w4-ca4642-kserve-6f45856c59-2xbqc\" (UID: \"80de7944-70f6-49e1-a0e8-4b82d5caad0d\") " pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" Nov 03 20:53:06.127850 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:06.127822 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80de7944-70f6-49e1-a0e8-4b82d5caad0d-dshm\") pod \"simulator-w4-ca4642-kserve-6f45856c59-2xbqc\" (UID: \"80de7944-70f6-49e1-a0e8-4b82d5caad0d\") " pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" Nov 03 20:53:06.128024 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:06.128008 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80de7944-70f6-49e1-a0e8-4b82d5caad0d-tls-certs\") pod \"simulator-w4-ca4642-kserve-6f45856c59-2xbqc\" (UID: \"80de7944-70f6-49e1-a0e8-4b82d5caad0d\") " pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" Nov 03 20:53:06.187189 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:06.187153 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lxcn\" (UniqueName: \"kubernetes.io/projected/80de7944-70f6-49e1-a0e8-4b82d5caad0d-kube-api-access-5lxcn\") pod \"simulator-w4-ca4642-kserve-6f45856c59-2xbqc\" (UID: \"80de7944-70f6-49e1-a0e8-4b82d5caad0d\") " pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" Nov 03 20:53:06.461929 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:06.461889 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" Nov 03 20:53:06.841608 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:06.841576 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc"] Nov 03 20:53:06.847234 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:53:06.847201 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod80de7944_70f6_49e1_a0e8_4b82d5caad0d.slice/crio-ee2293057a73bfdec3efd737f07315a1dc55a78c74dbcf1a4db0f40ec467af96 WatchSource:0}: Error finding container ee2293057a73bfdec3efd737f07315a1dc55a78c74dbcf1a4db0f40ec467af96: Status 404 returned error can't find the container with id ee2293057a73bfdec3efd737f07315a1dc55a78c74dbcf1a4db0f40ec467af96 Nov 03 20:53:06.962747 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:06.962712 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" event={"ID":"80de7944-70f6-49e1-a0e8-4b82d5caad0d","Type":"ContainerStarted","Data":"47f02a5ff2acc2619ef94f3e74a994157a324827f21ffd93323181a5d3576771"} Nov 03 20:53:06.962914 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:06.962757 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" event={"ID":"80de7944-70f6-49e1-a0e8-4b82d5caad0d","Type":"ContainerStarted","Data":"ee2293057a73bfdec3efd737f07315a1dc55a78c74dbcf1a4db0f40ec467af96"} Nov 03 20:53:07.079983 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:07.079916 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" podStartSLOduration=2.079900851 podStartE2EDuration="2.079900851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:07.079362335 +0000 UTC m=+822.304682155" watchObservedRunningTime="2026-11-03 20:53:07.079900851 +0000 UTC m=+822.305220641" Nov 03 20:53:07.462506 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:07.462468 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" Nov 03 20:53:16.323149 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:53:16.323108 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 20:53:16.463044 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:16.463003 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" Nov 03 20:53:16.475809 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:16.475781 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" Nov 03 20:53:17.032395 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:17.032363 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" Nov 03 20:53:29.387172 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:29.387135 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 20:53:42.949457 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:42.949355 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g"] Nov 03 20:53:42.953791 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:42.953768 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" Nov 03 20:53:43.006209 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:43.006165 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-89bb58\"/\"openshift-service-ca.crt\"" Nov 03 20:53:43.039214 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:43.039179 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1-dshm\") pod \"premium-w3-89bb58-kserve-59cdd84977-dmf9g\" (UID: \"7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1\") " pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" Nov 03 20:53:43.039399 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:43.039219 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1-model-cache\") pod \"premium-w3-89bb58-kserve-59cdd84977-dmf9g\" (UID: \"7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1\") " pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" Nov 03 20:53:43.039399 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:43.039248 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6hpg\" (UniqueName: \"kubernetes.io/projected/7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1-kube-api-access-c6hpg\") pod \"premium-w3-89bb58-kserve-59cdd84977-dmf9g\" (UID: \"7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1\") " pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" Nov 03 20:53:43.039528 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:43.039448 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1-tls-certs\") pod \"premium-w3-89bb58-kserve-59cdd84977-dmf9g\" (UID: \"7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1\") " pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" Nov 03 20:53:43.039666 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:43.039631 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1-tmp-dir\") pod \"premium-w3-89bb58-kserve-59cdd84977-dmf9g\" (UID: \"7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1\") " pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" Nov 03 20:53:43.039728 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:43.039690 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1-home\") pod \"premium-w3-89bb58-kserve-59cdd84977-dmf9g\" (UID: \"7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1\") " pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" Nov 03 20:53:43.055699 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:43.055667 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-89bb58\"/\"premium-w3-89bb58-kserve-self-signed-certs\"" Nov 03 20:53:43.055699 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:43.055708 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-89bb58\"/\"default-dockercfg-n6bg4\"" Nov 03 20:53:43.055923 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:43.055770 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-89bb58\"/\"kube-root-ca.crt\"" Nov 03 20:53:43.058574 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:43.058556 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g"] Nov 03 20:53:43.140575 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:43.140542 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1-dshm\") pod \"premium-w3-89bb58-kserve-59cdd84977-dmf9g\" (UID: \"7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1\") " pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" Nov 03 20:53:43.140575 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:43.140577 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1-model-cache\") pod \"premium-w3-89bb58-kserve-59cdd84977-dmf9g\" (UID: \"7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1\") " pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" Nov 03 20:53:43.140805 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:43.140593 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6hpg\" (UniqueName: \"kubernetes.io/projected/7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1-kube-api-access-c6hpg\") pod \"premium-w3-89bb58-kserve-59cdd84977-dmf9g\" (UID: \"7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1\") " pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" Nov 03 20:53:43.140805 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:43.140640 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1-tls-certs\") pod \"premium-w3-89bb58-kserve-59cdd84977-dmf9g\" (UID: \"7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1\") " pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" Nov 03 20:53:43.140805 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:43.140751 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1-tmp-dir\") pod \"premium-w3-89bb58-kserve-59cdd84977-dmf9g\" (UID: \"7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1\") " pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" Nov 03 20:53:43.140805 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:43.140778 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1-home\") pod \"premium-w3-89bb58-kserve-59cdd84977-dmf9g\" (UID: \"7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1\") " pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" Nov 03 20:53:43.141122 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:43.141097 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1-model-cache\") pod \"premium-w3-89bb58-kserve-59cdd84977-dmf9g\" (UID: \"7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1\") " pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" Nov 03 20:53:43.141122 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:43.141114 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1-home\") pod \"premium-w3-89bb58-kserve-59cdd84977-dmf9g\" (UID: \"7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1\") " pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" Nov 03 20:53:43.141224 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:43.141191 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1-tmp-dir\") pod \"premium-w3-89bb58-kserve-59cdd84977-dmf9g\" (UID: \"7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1\") " pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" Nov 03 20:53:43.143108 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:43.143088 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1-dshm\") pod \"premium-w3-89bb58-kserve-59cdd84977-dmf9g\" (UID: \"7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1\") " pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" Nov 03 20:53:43.143389 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:43.143373 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1-tls-certs\") pod \"premium-w3-89bb58-kserve-59cdd84977-dmf9g\" (UID: \"7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1\") " pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" Nov 03 20:53:43.202078 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:43.202009 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6hpg\" (UniqueName: \"kubernetes.io/projected/7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1-kube-api-access-c6hpg\") pod \"premium-w3-89bb58-kserve-59cdd84977-dmf9g\" (UID: \"7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1\") " pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" Nov 03 20:53:43.292866 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:43.292837 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" Nov 03 20:53:43.686929 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:43.686905 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g"] Nov 03 20:53:43.688259 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:53:43.688230 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f8e4e8b_bb78_4b3d_bc7d_2a35d9a05fb1.slice/crio-6262bc5f41af83ab2e2edd57e7f866a9da46f7705fddb8fda0aafa65a2414423 WatchSource:0}: Error finding container 6262bc5f41af83ab2e2edd57e7f866a9da46f7705fddb8fda0aafa65a2414423: Status 404 returned error can't find the container with id 6262bc5f41af83ab2e2edd57e7f866a9da46f7705fddb8fda0aafa65a2414423 Nov 03 20:53:44.189591 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:44.189553 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" event={"ID":"7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1","Type":"ContainerStarted","Data":"af324f45577a5dbeb6e03ba7f663ed039a97cc397d4d4995a69ac382110096ee"} Nov 03 20:53:44.189984 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:44.189601 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" event={"ID":"7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1","Type":"ContainerStarted","Data":"6262bc5f41af83ab2e2edd57e7f866a9da46f7705fddb8fda0aafa65a2414423"} Nov 03 20:53:44.293923 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:44.293887 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" Nov 03 20:53:44.314798 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:44.314733 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" podStartSLOduration=2.3147194349999998 podStartE2EDuration="2.314719435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:44.313758396 +0000 UTC m=+859.539078226" watchObservedRunningTime="2026-11-03 20:53:44.314719435 +0000 UTC m=+859.540039225" Nov 03 20:53:46.322410 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:53:46.322370 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 20:53:53.293503 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:53.293467 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" Nov 03 20:53:53.306169 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:53.306144 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" Nov 03 20:53:53.439280 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:53.439238 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 20:53:54.263359 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:53:54.263321 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" Nov 03 20:54:16.323357 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:54:16.323319 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 20:54:18.697679 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:18.697642 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 20:54:18.821371 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:18.821335 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq"] Nov 03 20:54:18.825577 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:18.825555 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" Nov 03 20:54:18.882375 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:18.882346 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-e22abf\"/\"openshift-service-ca.crt\"" Nov 03 20:54:18.895216 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:18.895189 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-e22abf\"/\"kube-root-ca.crt\"" Nov 03 20:54:18.930968 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:18.930934 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-e22abf\"/\"premium-w5-e22abf-kserve-self-signed-certs\"" Nov 03 20:54:18.930968 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:18.930959 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-e22abf\"/\"default-dockercfg-gdxxs\"" Nov 03 20:54:18.934487 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:18.934460 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq"] Nov 03 20:54:18.934984 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:18.934962 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3-home\") pod \"premium-w5-e22abf-kserve-f487d7c4c-sz9xq\" (UID: \"7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3\") " pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" Nov 03 20:54:18.935032 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:18.935000 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3-model-cache\") pod \"premium-w5-e22abf-kserve-f487d7c4c-sz9xq\" (UID: \"7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3\") " pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" Nov 03 20:54:18.935032 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:18.935027 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3-tmp-dir\") pod \"premium-w5-e22abf-kserve-f487d7c4c-sz9xq\" (UID: \"7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3\") " pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" Nov 03 20:54:18.935200 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:18.935179 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3-tls-certs\") pod \"premium-w5-e22abf-kserve-f487d7c4c-sz9xq\" (UID: \"7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3\") " pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" Nov 03 20:54:18.935272 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:18.935260 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3-dshm\") pod \"premium-w5-e22abf-kserve-f487d7c4c-sz9xq\" (UID: \"7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3\") " pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" Nov 03 20:54:18.935358 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:18.935342 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxpd4\" (UniqueName: \"kubernetes.io/projected/7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3-kube-api-access-pxpd4\") pod \"premium-w5-e22abf-kserve-f487d7c4c-sz9xq\" (UID: \"7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3\") " pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" Nov 03 20:54:19.036258 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:19.036175 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3-tls-certs\") pod \"premium-w5-e22abf-kserve-f487d7c4c-sz9xq\" (UID: \"7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3\") " pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" Nov 03 20:54:19.036258 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:19.036242 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3-dshm\") pod \"premium-w5-e22abf-kserve-f487d7c4c-sz9xq\" (UID: \"7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3\") " pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" Nov 03 20:54:19.036520 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:19.036306 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxpd4\" (UniqueName: \"kubernetes.io/projected/7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3-kube-api-access-pxpd4\") pod \"premium-w5-e22abf-kserve-f487d7c4c-sz9xq\" (UID: \"7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3\") " pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" Nov 03 20:54:19.036520 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:19.036331 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3-home\") pod \"premium-w5-e22abf-kserve-f487d7c4c-sz9xq\" (UID: \"7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3\") " pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" Nov 03 20:54:19.036520 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:19.036352 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3-model-cache\") pod \"premium-w5-e22abf-kserve-f487d7c4c-sz9xq\" (UID: \"7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3\") " pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" Nov 03 20:54:19.036520 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:19.036369 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3-tmp-dir\") pod \"premium-w5-e22abf-kserve-f487d7c4c-sz9xq\" (UID: \"7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3\") " pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" Nov 03 20:54:19.036818 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:19.036795 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3-model-cache\") pod \"premium-w5-e22abf-kserve-f487d7c4c-sz9xq\" (UID: \"7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3\") " pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" Nov 03 20:54:19.036891 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:19.036810 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3-tmp-dir\") pod \"premium-w5-e22abf-kserve-f487d7c4c-sz9xq\" (UID: \"7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3\") " pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" Nov 03 20:54:19.036891 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:19.036865 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3-home\") pod \"premium-w5-e22abf-kserve-f487d7c4c-sz9xq\" (UID: \"7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3\") " pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" Nov 03 20:54:19.038793 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:19.038773 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3-dshm\") pod \"premium-w5-e22abf-kserve-f487d7c4c-sz9xq\" (UID: \"7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3\") " pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" Nov 03 20:54:19.039062 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:19.039044 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3-tls-certs\") pod \"premium-w5-e22abf-kserve-f487d7c4c-sz9xq\" (UID: \"7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3\") " pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" Nov 03 20:54:19.099136 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:19.099101 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxpd4\" (UniqueName: \"kubernetes.io/projected/7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3-kube-api-access-pxpd4\") pod \"premium-w5-e22abf-kserve-f487d7c4c-sz9xq\" (UID: \"7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3\") " pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" Nov 03 20:54:19.138915 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:19.138877 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" Nov 03 20:54:19.521916 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:19.521878 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq"] Nov 03 20:54:19.524063 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:54:19.524037 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c05fc7b_83f3_43d0_830b_f4fc2e9fadb3.slice/crio-f931558a9ab1e734a5fff2b9dad210cecf516a4ca147663515c05b9c43024670 WatchSource:0}: Error finding container f931558a9ab1e734a5fff2b9dad210cecf516a4ca147663515c05b9c43024670: Status 404 returned error can't find the container with id f931558a9ab1e734a5fff2b9dad210cecf516a4ca147663515c05b9c43024670 Nov 03 20:54:20.406416 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:20.406379 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" event={"ID":"7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3","Type":"ContainerStarted","Data":"5722e751ffd83f95b108c793bf4e1583eeecd76bb01a8eb69c30a9d91c8d99b5"} Nov 03 20:54:20.406416 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:20.406445 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" event={"ID":"7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3","Type":"ContainerStarted","Data":"f931558a9ab1e734a5fff2b9dad210cecf516a4ca147663515c05b9c43024670"} Nov 03 20:54:20.530839 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:20.530784 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" podStartSLOduration=2.530768887 podStartE2EDuration="2.530768887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:54:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:54:20.52847963 +0000 UTC m=+895.753799442" watchObservedRunningTime="2026-11-03 20:54:20.530768887 +0000 UTC m=+895.756088677" Nov 03 20:54:21.139089 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:21.139050 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" Nov 03 20:54:29.140040 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:29.140004 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" Nov 03 20:54:29.153371 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:29.153342 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" Nov 03 20:54:29.470769 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:29.470733 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" Nov 03 20:54:42.756589 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:42.756560 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 20:54:43.231009 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:43.230974 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d0c9705-qv5tg"] Nov 03 20:54:43.253288 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:43.244801 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d0c9705-qv5tg" Nov 03 20:54:43.296090 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:43.296059 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d0c9705-qv5tg"] Nov 03 20:54:43.304709 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:43.304681 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:54:43.339959 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:43.339931 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77vtd\"" Nov 03 20:54:43.340154 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:43.339990 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:54:43.340154 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:43.340032 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:54:43.422192 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:43.422150 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcpck\" (UniqueName: \"kubernetes.io/projected/de93cd6a-03d9-4458-8fbb-1315f51890e0-kube-api-access-dcpck\") pod \"maas-api-revoke-sub-4d0c9705-qv5tg\" (UID: \"de93cd6a-03d9-4458-8fbb-1315f51890e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d0c9705-qv5tg" Nov 03 20:54:43.422392 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:43.422340 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de93cd6a-03d9-4458-8fbb-1315f51890e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d0c9705-qv5tg\" (UID: \"de93cd6a-03d9-4458-8fbb-1315f51890e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d0c9705-qv5tg" Nov 03 20:54:43.422472 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:43.422409 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de93cd6a-03d9-4458-8fbb-1315f51890e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d0c9705-qv5tg\" (UID: \"de93cd6a-03d9-4458-8fbb-1315f51890e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d0c9705-qv5tg" Nov 03 20:54:43.523599 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:43.523507 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de93cd6a-03d9-4458-8fbb-1315f51890e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d0c9705-qv5tg\" (UID: \"de93cd6a-03d9-4458-8fbb-1315f51890e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d0c9705-qv5tg" Nov 03 20:54:43.523599 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:43.523592 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de93cd6a-03d9-4458-8fbb-1315f51890e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d0c9705-qv5tg\" (UID: \"de93cd6a-03d9-4458-8fbb-1315f51890e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d0c9705-qv5tg" Nov 03 20:54:43.523836 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:43.523689 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcpck\" (UniqueName: \"kubernetes.io/projected/de93cd6a-03d9-4458-8fbb-1315f51890e0-kube-api-access-dcpck\") pod \"maas-api-revoke-sub-4d0c9705-qv5tg\" (UID: \"de93cd6a-03d9-4458-8fbb-1315f51890e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d0c9705-qv5tg" Nov 03 20:54:43.524279 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:43.524255 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de93cd6a-03d9-4458-8fbb-1315f51890e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d0c9705-qv5tg\" (UID: \"de93cd6a-03d9-4458-8fbb-1315f51890e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d0c9705-qv5tg" Nov 03 20:54:43.526090 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:43.526068 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de93cd6a-03d9-4458-8fbb-1315f51890e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d0c9705-qv5tg\" (UID: \"de93cd6a-03d9-4458-8fbb-1315f51890e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d0c9705-qv5tg" Nov 03 20:54:43.602553 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:43.602463 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcpck\" (UniqueName: \"kubernetes.io/projected/de93cd6a-03d9-4458-8fbb-1315f51890e0-kube-api-access-dcpck\") pod \"maas-api-revoke-sub-4d0c9705-qv5tg\" (UID: \"de93cd6a-03d9-4458-8fbb-1315f51890e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d0c9705-qv5tg" Nov 03 20:54:43.887921 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:43.887791 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d0c9705-qv5tg" Nov 03 20:54:44.296780 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:44.296740 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d0c9705-qv5tg"] Nov 03 20:54:44.301499 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:54:44.301466 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde93cd6a_03d9_4458_8fbb_1315f51890e0.slice/crio-2c45232a73110ccecfeeea20e26330bb64b65c7335638ed12b5d5760e248555d WatchSource:0}: Error finding container 2c45232a73110ccecfeeea20e26330bb64b65c7335638ed12b5d5760e248555d: Status 404 returned error can't find the container with id 2c45232a73110ccecfeeea20e26330bb64b65c7335638ed12b5d5760e248555d Nov 03 20:54:44.541979 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:44.541941 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d0c9705-qv5tg" event={"ID":"de93cd6a-03d9-4458-8fbb-1315f51890e0","Type":"ContainerStarted","Data":"2c45232a73110ccecfeeea20e26330bb64b65c7335638ed12b5d5760e248555d"} Nov 03 20:54:46.320864 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:54:46.320817 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 20:54:47.941863 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:47.941829 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr"] Nov 03 20:54:47.947333 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:47.947314 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr" Nov 03 20:54:47.976373 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:47.976339 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbb09c2d-a432-4507-a7d2-9e401f0e96bb-istio-podinfo\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr\" (UID: \"fbb09c2d-a432-4507-a7d2-9e401f0e96bb\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr" Nov 03 20:54:47.976548 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:47.976487 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7ngw\" (UniqueName: \"kubernetes.io/projected/fbb09c2d-a432-4507-a7d2-9e401f0e96bb-kube-api-access-x7ngw\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr\" (UID: \"fbb09c2d-a432-4507-a7d2-9e401f0e96bb\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr" Nov 03 20:54:47.976548 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:47.976528 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbb09c2d-a432-4507-a7d2-9e401f0e96bb-istio-envoy\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr\" (UID: \"fbb09c2d-a432-4507-a7d2-9e401f0e96bb\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr" Nov 03 20:54:47.976626 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:47.976565 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb09c2d-a432-4507-a7d2-9e401f0e96bb-workload-socket\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr\" (UID: \"fbb09c2d-a432-4507-a7d2-9e401f0e96bb\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr" Nov 03 20:54:47.976675 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:47.976662 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb09c2d-a432-4507-a7d2-9e401f0e96bb-credential-socket\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr\" (UID: \"fbb09c2d-a432-4507-a7d2-9e401f0e96bb\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr" Nov 03 20:54:47.976730 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:47.976719 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbb09c2d-a432-4507-a7d2-9e401f0e96bb-istio-data\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr\" (UID: \"fbb09c2d-a432-4507-a7d2-9e401f0e96bb\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr" Nov 03 20:54:47.976764 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:47.976749 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbb09c2d-a432-4507-a7d2-9e401f0e96bb-istiod-ca-cert\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr\" (UID: \"fbb09c2d-a432-4507-a7d2-9e401f0e96bb\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr" Nov 03 20:54:47.976889 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:47.976861 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbb09c2d-a432-4507-a7d2-9e401f0e96bb-istio-token\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr\" (UID: \"fbb09c2d-a432-4507-a7d2-9e401f0e96bb\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr" Nov 03 20:54:47.976961 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:47.976945 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbb09c2d-a432-4507-a7d2-9e401f0e96bb-workload-certs\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr\" (UID: \"fbb09c2d-a432-4507-a7d2-9e401f0e96bb\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr" Nov 03 20:54:47.999506 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:47.999474 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-12a423-openshift-default-dockercfg-2ztvn\"" Nov 03 20:54:48.051630 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:48.051598 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr"] Nov 03 20:54:48.077778 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:48.077744 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbb09c2d-a432-4507-a7d2-9e401f0e96bb-istio-podinfo\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr\" (UID: \"fbb09c2d-a432-4507-a7d2-9e401f0e96bb\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr" Nov 03 20:54:48.077952 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:48.077790 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7ngw\" (UniqueName: \"kubernetes.io/projected/fbb09c2d-a432-4507-a7d2-9e401f0e96bb-kube-api-access-x7ngw\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr\" (UID: \"fbb09c2d-a432-4507-a7d2-9e401f0e96bb\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr" Nov 03 20:54:48.077952 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:48.077835 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbb09c2d-a432-4507-a7d2-9e401f0e96bb-istio-envoy\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr\" (UID: \"fbb09c2d-a432-4507-a7d2-9e401f0e96bb\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr" Nov 03 20:54:48.077952 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:48.077866 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb09c2d-a432-4507-a7d2-9e401f0e96bb-workload-socket\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr\" (UID: \"fbb09c2d-a432-4507-a7d2-9e401f0e96bb\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr" Nov 03 20:54:48.078084 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:48.078063 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb09c2d-a432-4507-a7d2-9e401f0e96bb-credential-socket\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr\" (UID: \"fbb09c2d-a432-4507-a7d2-9e401f0e96bb\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr" Nov 03 20:54:48.078129 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:48.078106 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbb09c2d-a432-4507-a7d2-9e401f0e96bb-istio-data\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr\" (UID: \"fbb09c2d-a432-4507-a7d2-9e401f0e96bb\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr" Nov 03 20:54:48.078180 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:48.078154 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbb09c2d-a432-4507-a7d2-9e401f0e96bb-istiod-ca-cert\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr\" (UID: \"fbb09c2d-a432-4507-a7d2-9e401f0e96bb\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr" Nov 03 20:54:48.078258 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:48.078238 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb09c2d-a432-4507-a7d2-9e401f0e96bb-workload-socket\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr\" (UID: \"fbb09c2d-a432-4507-a7d2-9e401f0e96bb\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr" Nov 03 20:54:48.078322 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:48.078287 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbb09c2d-a432-4507-a7d2-9e401f0e96bb-istio-token\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr\" (UID: \"fbb09c2d-a432-4507-a7d2-9e401f0e96bb\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr" Nov 03 20:54:48.078379 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:48.078340 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbb09c2d-a432-4507-a7d2-9e401f0e96bb-workload-certs\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr\" (UID: \"fbb09c2d-a432-4507-a7d2-9e401f0e96bb\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr" Nov 03 20:54:48.078464 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:48.078420 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb09c2d-a432-4507-a7d2-9e401f0e96bb-credential-socket\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr\" (UID: \"fbb09c2d-a432-4507-a7d2-9e401f0e96bb\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr" Nov 03 20:54:48.078719 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:48.078697 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbb09c2d-a432-4507-a7d2-9e401f0e96bb-workload-certs\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr\" (UID: \"fbb09c2d-a432-4507-a7d2-9e401f0e96bb\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr" Nov 03 20:54:48.078719 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:48.078711 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbb09c2d-a432-4507-a7d2-9e401f0e96bb-istio-data\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr\" (UID: \"fbb09c2d-a432-4507-a7d2-9e401f0e96bb\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr" Nov 03 20:54:48.079212 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:48.079191 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbb09c2d-a432-4507-a7d2-9e401f0e96bb-istiod-ca-cert\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr\" (UID: \"fbb09c2d-a432-4507-a7d2-9e401f0e96bb\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr" Nov 03 20:54:48.080294 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:48.080273 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbb09c2d-a432-4507-a7d2-9e401f0e96bb-istio-envoy\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr\" (UID: \"fbb09c2d-a432-4507-a7d2-9e401f0e96bb\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr" Nov 03 20:54:48.080523 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:48.080505 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbb09c2d-a432-4507-a7d2-9e401f0e96bb-istio-podinfo\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr\" (UID: \"fbb09c2d-a432-4507-a7d2-9e401f0e96bb\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr" Nov 03 20:54:48.139140 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:48.139104 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbb09c2d-a432-4507-a7d2-9e401f0e96bb-istio-token\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr\" (UID: \"fbb09c2d-a432-4507-a7d2-9e401f0e96bb\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr" Nov 03 20:54:48.139289 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:48.139181 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7ngw\" (UniqueName: \"kubernetes.io/projected/fbb09c2d-a432-4507-a7d2-9e401f0e96bb-kube-api-access-x7ngw\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr\" (UID: \"fbb09c2d-a432-4507-a7d2-9e401f0e96bb\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr" Nov 03 20:54:48.261233 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:48.261146 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr" Nov 03 20:54:48.701881 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:48.701854 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr"] Nov 03 20:54:48.704178 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:54:48.704147 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbb09c2d_a432_4507_a7d2_9e401f0e96bb.slice/crio-c94c4e5ffe85e526f51e218f76e4629548e8149b0421204e528fc41bfc8ccbad WatchSource:0}: Error finding container c94c4e5ffe85e526f51e218f76e4629548e8149b0421204e528fc41bfc8ccbad: Status 404 returned error can't find the container with id c94c4e5ffe85e526f51e218f76e4629548e8149b0421204e528fc41bfc8ccbad Nov 03 20:54:48.707388 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:48.707355 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:54:48.707531 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:48.707422 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:54:48.707531 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:48.707482 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:54:49.584602 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:49.584559 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr" event={"ID":"fbb09c2d-a432-4507-a7d2-9e401f0e96bb","Type":"ContainerStarted","Data":"c3207900e035579b7be53a7527c6e070412de23241a06f4490a143d042b544e6"} Nov 03 20:54:49.585012 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:49.584689 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr" event={"ID":"fbb09c2d-a432-4507-a7d2-9e401f0e96bb","Type":"ContainerStarted","Data":"c94c4e5ffe85e526f51e218f76e4629548e8149b0421204e528fc41bfc8ccbad"} Nov 03 20:54:49.777411 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:49.777340 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr" podStartSLOduration=2.777317538 podStartE2EDuration="2.777317538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:54:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:54:49.706696633 +0000 UTC m=+924.932016436" watchObservedRunningTime="2026-11-03 20:54:49.777317538 +0000 UTC m=+925.002637329" Nov 03 20:54:50.276560 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:50.274494 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr" Nov 03 20:54:50.277536 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:50.277507 4305 patch_prober.go:29] interesting pod/e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 03 20:54:50.277680 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:50.277577 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr" podUID="fbb09c2d-a432-4507-a7d2-9e401f0e96bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 03 20:54:50.593562 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:50.593477 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d0c9705-qv5tg" event={"ID":"de93cd6a-03d9-4458-8fbb-1315f51890e0","Type":"ContainerStarted","Data":"78ce158ced8712488aa431a6f61b504347fc7656eba0c26026f89aa38ebcc2ca"} Nov 03 20:54:51.273469 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:51.272918 4305 patch_prober.go:29] interesting pod/e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 03 20:54:51.273469 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:51.273013 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr" podUID="fbb09c2d-a432-4507-a7d2-9e401f0e96bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 03 20:54:51.601938 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:51.601838 4305 generic.go:356] "Generic (PLEG): container finished" podID="de93cd6a-03d9-4458-8fbb-1315f51890e0" containerID="78ce158ced8712488aa431a6f61b504347fc7656eba0c26026f89aa38ebcc2ca" exitCode=0 Nov 03 20:54:51.602327 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:51.601926 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d0c9705-qv5tg" event={"ID":"de93cd6a-03d9-4458-8fbb-1315f51890e0","Type":"ContainerDied","Data":"78ce158ced8712488aa431a6f61b504347fc7656eba0c26026f89aa38ebcc2ca"} Nov 03 20:54:52.277014 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:52.276953 4305 patch_prober.go:29] interesting pod/e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 03 20:54:52.277223 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:52.277030 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr" podUID="fbb09c2d-a432-4507-a7d2-9e401f0e96bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 03 20:54:52.809552 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:52.809525 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d0c9705-qv5tg" Nov 03 20:54:52.932895 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:52.932860 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de93cd6a-03d9-4458-8fbb-1315f51890e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de93cd6a-03d9-4458-8fbb-1315f51890e0-maas-api-openshift-ca\") pod \"de93cd6a-03d9-4458-8fbb-1315f51890e0\" (UID: \"de93cd6a-03d9-4458-8fbb-1315f51890e0\") " Nov 03 20:54:52.933088 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:52.932919 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de93cd6a-03d9-4458-8fbb-1315f51890e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de93cd6a-03d9-4458-8fbb-1315f51890e0-maas-api-xks-ca\") pod \"de93cd6a-03d9-4458-8fbb-1315f51890e0\" (UID: \"de93cd6a-03d9-4458-8fbb-1315f51890e0\") " Nov 03 20:54:52.933088 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:52.933011 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de93cd6a-03d9-4458-8fbb-1315f51890e0-kube-api-access-dcpck\" (UniqueName: \"kubernetes.io/projected/de93cd6a-03d9-4458-8fbb-1315f51890e0-kube-api-access-dcpck\") pod \"de93cd6a-03d9-4458-8fbb-1315f51890e0\" (UID: \"de93cd6a-03d9-4458-8fbb-1315f51890e0\") " Nov 03 20:54:52.933310 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:52.933284 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de93cd6a-03d9-4458-8fbb-1315f51890e0-maas-api-openshift-ca" pod "de93cd6a-03d9-4458-8fbb-1315f51890e0" (UID: "de93cd6a-03d9-4458-8fbb-1315f51890e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:54:52.933636 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:52.933611 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de93cd6a-03d9-4458-8fbb-1315f51890e0-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:54:52.935289 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:52.935264 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de93cd6a-03d9-4458-8fbb-1315f51890e0-maas-api-xks-ca" pod "de93cd6a-03d9-4458-8fbb-1315f51890e0" (UID: "de93cd6a-03d9-4458-8fbb-1315f51890e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:54:52.935380 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:52.935362 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de93cd6a-03d9-4458-8fbb-1315f51890e0-kube-api-access-dcpck" pod "de93cd6a-03d9-4458-8fbb-1315f51890e0" (UID: "de93cd6a-03d9-4458-8fbb-1315f51890e0"). InnerVolumeSpecName "kube-api-access-dcpck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:54:53.034511 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:53.034475 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcpck\" (UniqueName: \"kubernetes.io/projected/de93cd6a-03d9-4458-8fbb-1315f51890e0-kube-api-access-dcpck\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:54:53.034511 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:53.034508 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de93cd6a-03d9-4458-8fbb-1315f51890e0-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:54:53.276715 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:53.276567 4305 patch_prober.go:29] interesting pod/e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 03 20:54:53.276715 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:53.276628 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr" podUID="fbb09c2d-a432-4507-a7d2-9e401f0e96bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 03 20:54:53.615667 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:53.615590 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d0c9705-qv5tg" Nov 03 20:54:53.615815 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:53.615593 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d0c9705-qv5tg" event={"ID":"de93cd6a-03d9-4458-8fbb-1315f51890e0","Type":"ContainerDied","Data":"2c45232a73110ccecfeeea20e26330bb64b65c7335638ed12b5d5760e248555d"} Nov 03 20:54:53.615815 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:53.615713 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c45232a73110ccecfeeea20e26330bb64b65c7335638ed12b5d5760e248555d" Nov 03 20:54:54.267506 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:54.261827 4305 patch_prober.go:29] interesting pod/e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 03 20:54:54.267506 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:54.261928 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr" podUID="fbb09c2d-a432-4507-a7d2-9e401f0e96bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 03 20:54:55.261683 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:55.261646 4305 patch_prober.go:29] interesting pod/e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Nov 03 20:54:55.261867 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:55.261714 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr" podUID="fbb09c2d-a432-4507-a7d2-9e401f0e96bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Nov 03 20:54:56.266553 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:56.266518 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr" Nov 03 20:54:56.266962 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:56.266790 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr" Nov 03 20:54:56.267543 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:54:56.267524 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr" Nov 03 20:55:16.322875 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:55:16.322837 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 20:55:16.743321 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:16.743287 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-931ba160-mgltm"] Nov 03 20:55:16.743846 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:16.743827 4305 cpu_manager.go:414] "containerMap: removing container" podUID="de93cd6a-03d9-4458-8fbb-1315f51890e0" containerName="revoke-keys" Nov 03 20:55:16.743916 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:16.743849 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de93cd6a-03d9-4458-8fbb-1315f51890e0" containerName="revoke-keys" Nov 03 20:55:16.743959 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:16.743955 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="de93cd6a-03d9-4458-8fbb-1315f51890e0" containerName="revoke-keys" Nov 03 20:55:16.747198 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:16.747181 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-931ba160-mgltm" Nov 03 20:55:16.847404 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:16.847368 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a10b3870-8ea5-4d05-8d81-c3061e45340e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-931ba160-mgltm\" (UID: \"a10b3870-8ea5-4d05-8d81-c3061e45340e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-931ba160-mgltm" Nov 03 20:55:16.847585 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:16.847484 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frs9r\" (UniqueName: \"kubernetes.io/projected/a10b3870-8ea5-4d05-8d81-c3061e45340e-kube-api-access-frs9r\") pod \"maas-api-revoke-sub-931ba160-mgltm\" (UID: \"a10b3870-8ea5-4d05-8d81-c3061e45340e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-931ba160-mgltm" Nov 03 20:55:16.847585 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:16.847546 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a10b3870-8ea5-4d05-8d81-c3061e45340e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-931ba160-mgltm\" (UID: \"a10b3870-8ea5-4d05-8d81-c3061e45340e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-931ba160-mgltm" Nov 03 20:55:16.856456 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:16.856397 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:55:16.856625 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:16.856454 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77vtd\"" Nov 03 20:55:16.856625 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:16.856538 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:55:16.856739 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:16.856728 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:55:16.862334 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:16.862291 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-931ba160-mgltm"] Nov 03 20:55:16.948853 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:16.948808 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a10b3870-8ea5-4d05-8d81-c3061e45340e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-931ba160-mgltm\" (UID: \"a10b3870-8ea5-4d05-8d81-c3061e45340e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-931ba160-mgltm" Nov 03 20:55:16.949080 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:16.948858 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frs9r\" (UniqueName: \"kubernetes.io/projected/a10b3870-8ea5-4d05-8d81-c3061e45340e-kube-api-access-frs9r\") pod \"maas-api-revoke-sub-931ba160-mgltm\" (UID: \"a10b3870-8ea5-4d05-8d81-c3061e45340e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-931ba160-mgltm" Nov 03 20:55:16.949080 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:16.948936 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a10b3870-8ea5-4d05-8d81-c3061e45340e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-931ba160-mgltm\" (UID: \"a10b3870-8ea5-4d05-8d81-c3061e45340e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-931ba160-mgltm" Nov 03 20:55:16.949574 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:16.949547 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a10b3870-8ea5-4d05-8d81-c3061e45340e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-931ba160-mgltm\" (UID: \"a10b3870-8ea5-4d05-8d81-c3061e45340e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-931ba160-mgltm" Nov 03 20:55:16.951409 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:16.951389 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a10b3870-8ea5-4d05-8d81-c3061e45340e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-931ba160-mgltm\" (UID: \"a10b3870-8ea5-4d05-8d81-c3061e45340e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-931ba160-mgltm" Nov 03 20:55:17.009842 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:17.009770 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frs9r\" (UniqueName: \"kubernetes.io/projected/a10b3870-8ea5-4d05-8d81-c3061e45340e-kube-api-access-frs9r\") pod \"maas-api-revoke-sub-931ba160-mgltm\" (UID: \"a10b3870-8ea5-4d05-8d81-c3061e45340e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-931ba160-mgltm" Nov 03 20:55:17.060955 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:17.060923 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-931ba160-mgltm" Nov 03 20:55:17.442956 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:17.442856 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-931ba160-mgltm"] Nov 03 20:55:17.446160 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:55:17.446126 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda10b3870_8ea5_4d05_8d81_c3061e45340e.slice/crio-63ccd75a95c051c999f71c26168acdfe8376be3866c0926c58f2ec832b102a4e WatchSource:0}: Error finding container 63ccd75a95c051c999f71c26168acdfe8376be3866c0926c58f2ec832b102a4e: Status 404 returned error can't find the container with id 63ccd75a95c051c999f71c26168acdfe8376be3866c0926c58f2ec832b102a4e Nov 03 20:55:17.762256 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:17.762224 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-931ba160-mgltm" event={"ID":"a10b3870-8ea5-4d05-8d81-c3061e45340e","Type":"ContainerStarted","Data":"63ccd75a95c051c999f71c26168acdfe8376be3866c0926c58f2ec832b102a4e"} Nov 03 20:55:18.769911 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:18.769881 4305 generic.go:356] "Generic (PLEG): container finished" podID="a10b3870-8ea5-4d05-8d81-c3061e45340e" containerID="bc4ea3368e5bf7d3eb45b0b4f1ba45fdb232ee51835d1deb8cbbf75cf5b3eb9c" exitCode=0 Nov 03 20:55:18.770313 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:18.769971 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-931ba160-mgltm" event={"ID":"a10b3870-8ea5-4d05-8d81-c3061e45340e","Type":"ContainerDied","Data":"bc4ea3368e5bf7d3eb45b0b4f1ba45fdb232ee51835d1deb8cbbf75cf5b3eb9c"} Nov 03 20:55:19.481945 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:19.481913 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn"] Nov 03 20:55:19.485892 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:19.485870 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn" Nov 03 20:55:19.579079 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:19.579024 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fea2251-d380-4c79-a754-1a3ec7bdf829-istio-data\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn\" (UID: \"2fea2251-d380-4c79-a754-1a3ec7bdf829\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn" Nov 03 20:55:19.579295 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:19.579135 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fea2251-d380-4c79-a754-1a3ec7bdf829-workload-certs\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn\" (UID: \"2fea2251-d380-4c79-a754-1a3ec7bdf829\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn" Nov 03 20:55:19.579295 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:19.579161 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fea2251-d380-4c79-a754-1a3ec7bdf829-credential-socket\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn\" (UID: \"2fea2251-d380-4c79-a754-1a3ec7bdf829\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn" Nov 03 20:55:19.579295 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:19.579258 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fea2251-d380-4c79-a754-1a3ec7bdf829-istio-envoy\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn\" (UID: \"2fea2251-d380-4c79-a754-1a3ec7bdf829\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn" Nov 03 20:55:19.579494 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:19.579338 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fea2251-d380-4c79-a754-1a3ec7bdf829-istio-token\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn\" (UID: \"2fea2251-d380-4c79-a754-1a3ec7bdf829\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn" Nov 03 20:55:19.579494 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:19.579360 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fea2251-d380-4c79-a754-1a3ec7bdf829-istiod-ca-cert\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn\" (UID: \"2fea2251-d380-4c79-a754-1a3ec7bdf829\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn" Nov 03 20:55:19.579494 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:19.579428 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qjnt\" (UniqueName: \"kubernetes.io/projected/2fea2251-d380-4c79-a754-1a3ec7bdf829-kube-api-access-9qjnt\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn\" (UID: \"2fea2251-d380-4c79-a754-1a3ec7bdf829\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn" Nov 03 20:55:19.579494 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:19.579470 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fea2251-d380-4c79-a754-1a3ec7bdf829-workload-socket\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn\" (UID: \"2fea2251-d380-4c79-a754-1a3ec7bdf829\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn" Nov 03 20:55:19.579717 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:19.579524 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fea2251-d380-4c79-a754-1a3ec7bdf829-istio-podinfo\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn\" (UID: \"2fea2251-d380-4c79-a754-1a3ec7bdf829\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn" Nov 03 20:55:19.583276 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:19.583250 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-862138-openshift-default-dockercfg-8dw8p\"" Nov 03 20:55:19.587283 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:19.587258 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn"] Nov 03 20:55:19.667972 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:19.667936 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 20:55:19.680775 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:19.680737 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fea2251-d380-4c79-a754-1a3ec7bdf829-istio-data\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn\" (UID: \"2fea2251-d380-4c79-a754-1a3ec7bdf829\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn" Nov 03 20:55:19.680946 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:19.680802 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fea2251-d380-4c79-a754-1a3ec7bdf829-workload-certs\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn\" (UID: \"2fea2251-d380-4c79-a754-1a3ec7bdf829\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn" Nov 03 20:55:19.680946 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:19.680837 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fea2251-d380-4c79-a754-1a3ec7bdf829-credential-socket\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn\" (UID: \"2fea2251-d380-4c79-a754-1a3ec7bdf829\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn" Nov 03 20:55:19.680946 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:19.680863 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fea2251-d380-4c79-a754-1a3ec7bdf829-istio-envoy\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn\" (UID: \"2fea2251-d380-4c79-a754-1a3ec7bdf829\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn" Nov 03 20:55:19.680946 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:19.680911 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fea2251-d380-4c79-a754-1a3ec7bdf829-istio-token\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn\" (UID: \"2fea2251-d380-4c79-a754-1a3ec7bdf829\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn" Nov 03 20:55:19.680946 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:19.680941 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fea2251-d380-4c79-a754-1a3ec7bdf829-istiod-ca-cert\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn\" (UID: \"2fea2251-d380-4c79-a754-1a3ec7bdf829\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn" Nov 03 20:55:19.681194 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:19.681015 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qjnt\" (UniqueName: \"kubernetes.io/projected/2fea2251-d380-4c79-a754-1a3ec7bdf829-kube-api-access-9qjnt\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn\" (UID: \"2fea2251-d380-4c79-a754-1a3ec7bdf829\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn" Nov 03 20:55:19.681194 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:19.681046 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fea2251-d380-4c79-a754-1a3ec7bdf829-workload-socket\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn\" (UID: \"2fea2251-d380-4c79-a754-1a3ec7bdf829\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn" Nov 03 20:55:19.681194 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:19.681072 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fea2251-d380-4c79-a754-1a3ec7bdf829-istio-podinfo\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn\" (UID: \"2fea2251-d380-4c79-a754-1a3ec7bdf829\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn" Nov 03 20:55:19.681194 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:19.681185 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fea2251-d380-4c79-a754-1a3ec7bdf829-istio-data\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn\" (UID: \"2fea2251-d380-4c79-a754-1a3ec7bdf829\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn" Nov 03 20:55:19.681368 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:19.681268 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fea2251-d380-4c79-a754-1a3ec7bdf829-credential-socket\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn\" (UID: \"2fea2251-d380-4c79-a754-1a3ec7bdf829\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn" Nov 03 20:55:19.681481 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:19.681428 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fea2251-d380-4c79-a754-1a3ec7bdf829-workload-certs\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn\" (UID: \"2fea2251-d380-4c79-a754-1a3ec7bdf829\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn" Nov 03 20:55:19.681613 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:19.681591 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fea2251-d380-4c79-a754-1a3ec7bdf829-workload-socket\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn\" (UID: \"2fea2251-d380-4c79-a754-1a3ec7bdf829\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn" Nov 03 20:55:19.681778 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:19.681759 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fea2251-d380-4c79-a754-1a3ec7bdf829-istiod-ca-cert\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn\" (UID: \"2fea2251-d380-4c79-a754-1a3ec7bdf829\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn" Nov 03 20:55:19.683565 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:19.683543 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fea2251-d380-4c79-a754-1a3ec7bdf829-istio-envoy\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn\" (UID: \"2fea2251-d380-4c79-a754-1a3ec7bdf829\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn" Nov 03 20:55:19.683783 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:19.683763 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fea2251-d380-4c79-a754-1a3ec7bdf829-istio-podinfo\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn\" (UID: \"2fea2251-d380-4c79-a754-1a3ec7bdf829\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn" Nov 03 20:55:19.748189 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:19.748118 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qjnt\" (UniqueName: \"kubernetes.io/projected/2fea2251-d380-4c79-a754-1a3ec7bdf829-kube-api-access-9qjnt\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn\" (UID: \"2fea2251-d380-4c79-a754-1a3ec7bdf829\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn" Nov 03 20:55:19.752813 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:19.752790 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fea2251-d380-4c79-a754-1a3ec7bdf829-istio-token\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn\" (UID: \"2fea2251-d380-4c79-a754-1a3ec7bdf829\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn" Nov 03 20:55:19.800303 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:19.800270 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn" Nov 03 20:55:19.924950 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:19.924920 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-931ba160-mgltm" Nov 03 20:55:20.087012 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:20.086930 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a10b3870-8ea5-4d05-8d81-c3061e45340e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a10b3870-8ea5-4d05-8d81-c3061e45340e-maas-api-openshift-ca\") pod \"a10b3870-8ea5-4d05-8d81-c3061e45340e\" (UID: \"a10b3870-8ea5-4d05-8d81-c3061e45340e\") " Nov 03 20:55:20.087166 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:20.087042 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a10b3870-8ea5-4d05-8d81-c3061e45340e-kube-api-access-frs9r\" (UniqueName: \"kubernetes.io/projected/a10b3870-8ea5-4d05-8d81-c3061e45340e-kube-api-access-frs9r\") pod \"a10b3870-8ea5-4d05-8d81-c3061e45340e\" (UID: \"a10b3870-8ea5-4d05-8d81-c3061e45340e\") " Nov 03 20:55:20.087166 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:20.087092 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a10b3870-8ea5-4d05-8d81-c3061e45340e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a10b3870-8ea5-4d05-8d81-c3061e45340e-maas-api-xks-ca\") pod \"a10b3870-8ea5-4d05-8d81-c3061e45340e\" (UID: \"a10b3870-8ea5-4d05-8d81-c3061e45340e\") " Nov 03 20:55:20.087286 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:20.087263 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a10b3870-8ea5-4d05-8d81-c3061e45340e-maas-api-openshift-ca" pod "a10b3870-8ea5-4d05-8d81-c3061e45340e" (UID: "a10b3870-8ea5-4d05-8d81-c3061e45340e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:55:20.087512 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:20.087491 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a10b3870-8ea5-4d05-8d81-c3061e45340e-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:55:20.089285 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:20.089262 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a10b3870-8ea5-4d05-8d81-c3061e45340e-maas-api-xks-ca" pod "a10b3870-8ea5-4d05-8d81-c3061e45340e" (UID: "a10b3870-8ea5-4d05-8d81-c3061e45340e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:55:20.089362 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:20.089298 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a10b3870-8ea5-4d05-8d81-c3061e45340e-kube-api-access-frs9r" pod "a10b3870-8ea5-4d05-8d81-c3061e45340e" (UID: "a10b3870-8ea5-4d05-8d81-c3061e45340e"). InnerVolumeSpecName "kube-api-access-frs9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:55:20.188584 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:20.188556 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frs9r\" (UniqueName: \"kubernetes.io/projected/a10b3870-8ea5-4d05-8d81-c3061e45340e-kube-api-access-frs9r\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:55:20.188584 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:20.188580 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a10b3870-8ea5-4d05-8d81-c3061e45340e-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:55:20.242235 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:20.242208 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn"] Nov 03 20:55:20.243424 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:55:20.243399 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fea2251_d380_4c79_a754_1a3ec7bdf829.slice/crio-ba31d07d3ff03871282fc2e775febd2637614fa930d20f55d233be381d08782a WatchSource:0}: Error finding container ba31d07d3ff03871282fc2e775febd2637614fa930d20f55d233be381d08782a: Status 404 returned error can't find the container with id ba31d07d3ff03871282fc2e775febd2637614fa930d20f55d233be381d08782a Nov 03 20:55:20.245924 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:20.245890 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:55:20.246033 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:20.245968 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:55:20.246033 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:20.246022 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:55:20.784872 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:20.784791 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn" event={"ID":"2fea2251-d380-4c79-a754-1a3ec7bdf829","Type":"ContainerStarted","Data":"caa40f2d41760bebe3ffe2434ec0064e45f551b4431a64c9fd2351ec5f854170"} Nov 03 20:55:20.784872 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:20.784858 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn" event={"ID":"2fea2251-d380-4c79-a754-1a3ec7bdf829","Type":"ContainerStarted","Data":"ba31d07d3ff03871282fc2e775febd2637614fa930d20f55d233be381d08782a"} Nov 03 20:55:20.787488 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:20.787427 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-931ba160-mgltm" event={"ID":"a10b3870-8ea5-4d05-8d81-c3061e45340e","Type":"ContainerDied","Data":"63ccd75a95c051c999f71c26168acdfe8376be3866c0926c58f2ec832b102a4e"} Nov 03 20:55:20.787637 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:20.787507 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63ccd75a95c051c999f71c26168acdfe8376be3866c0926c58f2ec832b102a4e" Nov 03 20:55:20.787637 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:20.787473 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-931ba160-mgltm" Nov 03 20:55:20.800651 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:20.800612 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn" Nov 03 20:55:20.911142 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:20.911084 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn" podStartSLOduration=1.9110688200000001 podStartE2EDuration="1.91106882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:55:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:55:20.906546791 +0000 UTC m=+956.131866581" watchObservedRunningTime="2026-11-03 20:55:20.91106882 +0000 UTC m=+956.136388609" Nov 03 20:55:21.802336 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:21.802297 4305 patch_prober.go:29] interesting pod/e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 03 20:55:21.802910 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:21.802356 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn" podUID="2fea2251-d380-4c79-a754-1a3ec7bdf829" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 03 20:55:22.801086 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:22.801055 4305 patch_prober.go:29] interesting pod/e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 03 20:55:22.801279 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:22.801106 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn" podUID="2fea2251-d380-4c79-a754-1a3ec7bdf829" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 03 20:55:23.693657 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:23.693619 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c77973b3-ngzpp"] Nov 03 20:55:23.694168 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:23.694153 4305 cpu_manager.go:414] "containerMap: removing container" podUID="a10b3870-8ea5-4d05-8d81-c3061e45340e" containerName="revoke-keys" Nov 03 20:55:23.694213 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:23.694169 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a10b3870-8ea5-4d05-8d81-c3061e45340e" containerName="revoke-keys" Nov 03 20:55:23.694276 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:23.694268 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="a10b3870-8ea5-4d05-8d81-c3061e45340e" containerName="revoke-keys" Nov 03 20:55:23.700996 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:23.700962 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c77973b3-ngzpp" Nov 03 20:55:23.768459 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:23.766748 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c77973b3-ngzpp"] Nov 03 20:55:23.800726 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:23.800694 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:55:23.800726 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:23.800722 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:55:23.800979 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:23.800764 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77vtd\"" Nov 03 20:55:23.801090 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:23.801068 4305 patch_prober.go:29] interesting pod/e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 03 20:55:23.801159 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:23.801136 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn" podUID="2fea2251-d380-4c79-a754-1a3ec7bdf829" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 03 20:55:23.805475 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:23.805423 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:55:23.836767 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:23.836735 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8b10daa-ff51-4e8d-96e3-93e5c1740197-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c77973b3-ngzpp\" (UID: \"f8b10daa-ff51-4e8d-96e3-93e5c1740197\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c77973b3-ngzpp" Nov 03 20:55:23.836953 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:23.836864 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpgng\" (UniqueName: \"kubernetes.io/projected/f8b10daa-ff51-4e8d-96e3-93e5c1740197-kube-api-access-xpgng\") pod \"maas-api-revoke-sub-c77973b3-ngzpp\" (UID: \"f8b10daa-ff51-4e8d-96e3-93e5c1740197\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c77973b3-ngzpp" Nov 03 20:55:23.837040 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:23.837018 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8b10daa-ff51-4e8d-96e3-93e5c1740197-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c77973b3-ngzpp\" (UID: \"f8b10daa-ff51-4e8d-96e3-93e5c1740197\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c77973b3-ngzpp" Nov 03 20:55:23.938647 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:23.938603 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpgng\" (UniqueName: \"kubernetes.io/projected/f8b10daa-ff51-4e8d-96e3-93e5c1740197-kube-api-access-xpgng\") pod \"maas-api-revoke-sub-c77973b3-ngzpp\" (UID: \"f8b10daa-ff51-4e8d-96e3-93e5c1740197\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c77973b3-ngzpp" Nov 03 20:55:23.938853 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:23.938777 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8b10daa-ff51-4e8d-96e3-93e5c1740197-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c77973b3-ngzpp\" (UID: \"f8b10daa-ff51-4e8d-96e3-93e5c1740197\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c77973b3-ngzpp" Nov 03 20:55:23.939053 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:23.939023 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8b10daa-ff51-4e8d-96e3-93e5c1740197-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c77973b3-ngzpp\" (UID: \"f8b10daa-ff51-4e8d-96e3-93e5c1740197\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c77973b3-ngzpp" Nov 03 20:55:23.939793 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:23.939767 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8b10daa-ff51-4e8d-96e3-93e5c1740197-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c77973b3-ngzpp\" (UID: \"f8b10daa-ff51-4e8d-96e3-93e5c1740197\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c77973b3-ngzpp" Nov 03 20:55:23.941294 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:23.941271 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8b10daa-ff51-4e8d-96e3-93e5c1740197-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c77973b3-ngzpp\" (UID: \"f8b10daa-ff51-4e8d-96e3-93e5c1740197\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c77973b3-ngzpp" Nov 03 20:55:24.001463 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:24.001356 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpgng\" (UniqueName: \"kubernetes.io/projected/f8b10daa-ff51-4e8d-96e3-93e5c1740197-kube-api-access-xpgng\") pod \"maas-api-revoke-sub-c77973b3-ngzpp\" (UID: \"f8b10daa-ff51-4e8d-96e3-93e5c1740197\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c77973b3-ngzpp" Nov 03 20:55:24.015232 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:24.015207 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c77973b3-ngzpp" Nov 03 20:55:24.413895 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:24.413870 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c77973b3-ngzpp"] Nov 03 20:55:24.415160 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:55:24.415128 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8b10daa_ff51_4e8d_96e3_93e5c1740197.slice/crio-4be0a116e158cd4bd1594c682058c22f7d7e3ae0065b8299b4b4559c8fb82729 WatchSource:0}: Error finding container 4be0a116e158cd4bd1594c682058c22f7d7e3ae0065b8299b4b4559c8fb82729: Status 404 returned error can't find the container with id 4be0a116e158cd4bd1594c682058c22f7d7e3ae0065b8299b4b4559c8fb82729 Nov 03 20:55:24.801451 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:24.801406 4305 patch_prober.go:29] interesting pod/e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 03 20:55:24.801793 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:24.801487 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn" podUID="2fea2251-d380-4c79-a754-1a3ec7bdf829" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 03 20:55:24.814337 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:24.814310 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c77973b3-ngzpp" event={"ID":"f8b10daa-ff51-4e8d-96e3-93e5c1740197","Type":"ContainerStarted","Data":"86b9f07108518c07b99ae7e0e4870772ab608287c789bec95617e7e23057206b"} Nov 03 20:55:24.814496 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:24.814367 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c77973b3-ngzpp" event={"ID":"f8b10daa-ff51-4e8d-96e3-93e5c1740197","Type":"ContainerStarted","Data":"4be0a116e158cd4bd1594c682058c22f7d7e3ae0065b8299b4b4559c8fb82729"} Nov 03 20:55:24.986536 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:24.986461 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c77973b3-ngzpp" podStartSLOduration=1.9864247339999999 podStartE2EDuration="1.986424734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:55:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:55:24.95121714 +0000 UTC m=+960.176536932" watchObservedRunningTime="2026-11-03 20:55:24.986424734 +0000 UTC m=+960.211744526" Nov 03 20:55:25.801419 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:25.801333 4305 patch_prober.go:29] interesting pod/e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 03 20:55:25.801419 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:25.801403 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn" podUID="2fea2251-d380-4c79-a754-1a3ec7bdf829" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 03 20:55:25.821235 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:25.821203 4305 generic.go:356] "Generic (PLEG): container finished" podID="f8b10daa-ff51-4e8d-96e3-93e5c1740197" containerID="86b9f07108518c07b99ae7e0e4870772ab608287c789bec95617e7e23057206b" exitCode=0 Nov 03 20:55:25.821403 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:25.821293 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c77973b3-ngzpp" event={"ID":"f8b10daa-ff51-4e8d-96e3-93e5c1740197","Type":"ContainerDied","Data":"86b9f07108518c07b99ae7e0e4870772ab608287c789bec95617e7e23057206b"} Nov 03 20:55:26.801108 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:26.801075 4305 patch_prober.go:29] interesting pod/e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 03 20:55:26.801294 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:26.801140 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn" podUID="2fea2251-d380-4c79-a754-1a3ec7bdf829" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 03 20:55:26.974727 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:26.974704 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c77973b3-ngzpp" Nov 03 20:55:27.100489 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:27.100380 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8b10daa-ff51-4e8d-96e3-93e5c1740197-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8b10daa-ff51-4e8d-96e3-93e5c1740197-maas-api-xks-ca\") pod \"f8b10daa-ff51-4e8d-96e3-93e5c1740197\" (UID: \"f8b10daa-ff51-4e8d-96e3-93e5c1740197\") " Nov 03 20:55:27.100962 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:27.100938 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8b10daa-ff51-4e8d-96e3-93e5c1740197-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8b10daa-ff51-4e8d-96e3-93e5c1740197-maas-api-openshift-ca\") pod \"f8b10daa-ff51-4e8d-96e3-93e5c1740197\" (UID: \"f8b10daa-ff51-4e8d-96e3-93e5c1740197\") " Nov 03 20:55:27.101501 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:27.101483 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8b10daa-ff51-4e8d-96e3-93e5c1740197-kube-api-access-xpgng\" (UniqueName: \"kubernetes.io/projected/f8b10daa-ff51-4e8d-96e3-93e5c1740197-kube-api-access-xpgng\") pod \"f8b10daa-ff51-4e8d-96e3-93e5c1740197\" (UID: \"f8b10daa-ff51-4e8d-96e3-93e5c1740197\") " Nov 03 20:55:27.102464 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:27.101413 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8b10daa-ff51-4e8d-96e3-93e5c1740197-maas-api-openshift-ca" pod "f8b10daa-ff51-4e8d-96e3-93e5c1740197" (UID: "f8b10daa-ff51-4e8d-96e3-93e5c1740197"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:55:27.104953 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:27.104922 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8b10daa-ff51-4e8d-96e3-93e5c1740197-kube-api-access-xpgng" pod "f8b10daa-ff51-4e8d-96e3-93e5c1740197" (UID: "f8b10daa-ff51-4e8d-96e3-93e5c1740197"). InnerVolumeSpecName "kube-api-access-xpgng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:55:27.117839 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:27.117806 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8b10daa-ff51-4e8d-96e3-93e5c1740197-maas-api-xks-ca" pod "f8b10daa-ff51-4e8d-96e3-93e5c1740197" (UID: "f8b10daa-ff51-4e8d-96e3-93e5c1740197"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:55:27.203792 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:27.203760 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8b10daa-ff51-4e8d-96e3-93e5c1740197-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:55:27.203792 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:27.203795 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpgng\" (UniqueName: \"kubernetes.io/projected/f8b10daa-ff51-4e8d-96e3-93e5c1740197-kube-api-access-xpgng\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:55:27.204025 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:27.203811 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8b10daa-ff51-4e8d-96e3-93e5c1740197-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:55:27.805586 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:27.805558 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn" Nov 03 20:55:27.805834 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:27.805818 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn" Nov 03 20:55:27.806499 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:27.806479 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn" Nov 03 20:55:27.834412 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:27.834384 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c77973b3-ngzpp" Nov 03 20:55:27.834594 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:27.834419 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c77973b3-ngzpp" event={"ID":"f8b10daa-ff51-4e8d-96e3-93e5c1740197","Type":"ContainerDied","Data":"4be0a116e158cd4bd1594c682058c22f7d7e3ae0065b8299b4b4559c8fb82729"} Nov 03 20:55:27.834594 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:27.834489 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4be0a116e158cd4bd1594c682058c22f7d7e3ae0065b8299b4b4559c8fb82729" Nov 03 20:55:32.529293 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:32.529259 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a58fa67a-s5r5h"] Nov 03 20:55:32.529840 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:32.529828 4305 cpu_manager.go:414] "containerMap: removing container" podUID="f8b10daa-ff51-4e8d-96e3-93e5c1740197" containerName="revoke-keys" Nov 03 20:55:32.529911 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:32.529842 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8b10daa-ff51-4e8d-96e3-93e5c1740197" containerName="revoke-keys" Nov 03 20:55:32.529970 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:32.529961 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8b10daa-ff51-4e8d-96e3-93e5c1740197" containerName="revoke-keys" Nov 03 20:55:32.533591 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:32.533568 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a58fa67a-s5r5h" Nov 03 20:55:32.567297 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:32.567261 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9nmr\" (UniqueName: \"kubernetes.io/projected/b6b6b70b-9fc3-4786-bf43-afedba0e31a3-kube-api-access-t9nmr\") pod \"maas-api-revoke-sub-a58fa67a-s5r5h\" (UID: \"b6b6b70b-9fc3-4786-bf43-afedba0e31a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a58fa67a-s5r5h" Nov 03 20:55:32.567502 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:32.567316 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6b6b70b-9fc3-4786-bf43-afedba0e31a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a58fa67a-s5r5h\" (UID: \"b6b6b70b-9fc3-4786-bf43-afedba0e31a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a58fa67a-s5r5h" Nov 03 20:55:32.567802 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:32.567775 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6b6b70b-9fc3-4786-bf43-afedba0e31a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a58fa67a-s5r5h\" (UID: \"b6b6b70b-9fc3-4786-bf43-afedba0e31a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a58fa67a-s5r5h" Nov 03 20:55:32.593742 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:32.593691 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:55:32.635977 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:32.635941 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77vtd\"" Nov 03 20:55:32.636169 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:32.636000 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:55:32.636169 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:32.636068 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:55:32.639419 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:32.639397 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a58fa67a-s5r5h"] Nov 03 20:55:32.668594 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:32.668559 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6b6b70b-9fc3-4786-bf43-afedba0e31a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a58fa67a-s5r5h\" (UID: \"b6b6b70b-9fc3-4786-bf43-afedba0e31a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a58fa67a-s5r5h" Nov 03 20:55:32.668778 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:32.668650 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9nmr\" (UniqueName: \"kubernetes.io/projected/b6b6b70b-9fc3-4786-bf43-afedba0e31a3-kube-api-access-t9nmr\") pod \"maas-api-revoke-sub-a58fa67a-s5r5h\" (UID: \"b6b6b70b-9fc3-4786-bf43-afedba0e31a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a58fa67a-s5r5h" Nov 03 20:55:32.668778 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:32.668698 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6b6b70b-9fc3-4786-bf43-afedba0e31a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a58fa67a-s5r5h\" (UID: \"b6b6b70b-9fc3-4786-bf43-afedba0e31a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a58fa67a-s5r5h" Nov 03 20:55:32.669342 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:32.669319 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6b6b70b-9fc3-4786-bf43-afedba0e31a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a58fa67a-s5r5h\" (UID: \"b6b6b70b-9fc3-4786-bf43-afedba0e31a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a58fa67a-s5r5h" Nov 03 20:55:32.671231 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:32.671210 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6b6b70b-9fc3-4786-bf43-afedba0e31a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a58fa67a-s5r5h\" (UID: \"b6b6b70b-9fc3-4786-bf43-afedba0e31a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a58fa67a-s5r5h" Nov 03 20:55:32.729889 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:32.729861 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9nmr\" (UniqueName: \"kubernetes.io/projected/b6b6b70b-9fc3-4786-bf43-afedba0e31a3-kube-api-access-t9nmr\") pod \"maas-api-revoke-sub-a58fa67a-s5r5h\" (UID: \"b6b6b70b-9fc3-4786-bf43-afedba0e31a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a58fa67a-s5r5h" Nov 03 20:55:32.846915 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:32.846837 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a58fa67a-s5r5h" Nov 03 20:55:33.223414 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:33.223382 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a58fa67a-s5r5h"] Nov 03 20:55:33.224921 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:55:33.224890 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6b6b70b_9fc3_4786_bf43_afedba0e31a3.slice/crio-c6435dc1e0b42971a061c7f70a8cf8d332a6fdf46dc95f417a58f5e52929b702 WatchSource:0}: Error finding container c6435dc1e0b42971a061c7f70a8cf8d332a6fdf46dc95f417a58f5e52929b702: Status 404 returned error can't find the container with id c6435dc1e0b42971a061c7f70a8cf8d332a6fdf46dc95f417a58f5e52929b702 Nov 03 20:55:33.874970 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:33.874881 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a58fa67a-s5r5h" event={"ID":"b6b6b70b-9fc3-4786-bf43-afedba0e31a3","Type":"ContainerStarted","Data":"3828920ba73658dcc620bca4fae8d2dabb522f981dc05db247e94cd1ea233791"} Nov 03 20:55:33.874970 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:33.874930 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a58fa67a-s5r5h" event={"ID":"b6b6b70b-9fc3-4786-bf43-afedba0e31a3","Type":"ContainerStarted","Data":"c6435dc1e0b42971a061c7f70a8cf8d332a6fdf46dc95f417a58f5e52929b702"} Nov 03 20:55:33.998238 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:33.998191 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a58fa67a-s5r5h" podStartSLOduration=1.9981776610000002 podStartE2EDuration="1.998177661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:55:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:55:33.994988892 +0000 UTC m=+969.220308682" watchObservedRunningTime="2026-11-03 20:55:33.998177661 +0000 UTC m=+969.223497496" Nov 03 20:55:34.170076 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:34.170040 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jccsf"] Nov 03 20:55:34.170410 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:34.170273 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jccsf" podUID="bc5b2941-a16d-4ab5-a36b-12a7f381c8c0" containerName="istio-proxy" containerID="cri-o://a7c6681dd4637d18897540b6f35b4bc5d27fdab63018f303004341edd08f427b" gracePeriod=30 Nov 03 20:55:34.173878 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:34.173854 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7"] Nov 03 20:55:34.174065 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:34.174047 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7" podUID="39ce2073-d106-4204-be5d-9ea93f510db8" containerName="istio-proxy" containerID="cri-o://95f0c87e9c27adfd0113034e263a9cdd1a09ae70aee578bb2728fe2769424624" gracePeriod=30 Nov 03 20:55:34.500778 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:34.500701 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2f65010-2rf7k"] Nov 03 20:55:34.504722 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:34.504702 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2f65010-2rf7k" Nov 03 20:55:34.562864 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:34.562834 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2f65010-2rf7k"] Nov 03 20:55:34.588071 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:34.588023 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jvdl\" (UniqueName: \"kubernetes.io/projected/4f7a55ca-e2b1-469e-b5fa-22d4558de0fb-kube-api-access-2jvdl\") pod \"maas-api-revoke-sub-c2f65010-2rf7k\" (UID: \"4f7a55ca-e2b1-469e-b5fa-22d4558de0fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2f65010-2rf7k" Nov 03 20:55:34.588283 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:34.588143 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f7a55ca-e2b1-469e-b5fa-22d4558de0fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2f65010-2rf7k\" (UID: \"4f7a55ca-e2b1-469e-b5fa-22d4558de0fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2f65010-2rf7k" Nov 03 20:55:34.588283 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:34.588200 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f7a55ca-e2b1-469e-b5fa-22d4558de0fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2f65010-2rf7k\" (UID: \"4f7a55ca-e2b1-469e-b5fa-22d4558de0fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2f65010-2rf7k" Nov 03 20:55:34.688950 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:34.688915 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jvdl\" (UniqueName: \"kubernetes.io/projected/4f7a55ca-e2b1-469e-b5fa-22d4558de0fb-kube-api-access-2jvdl\") pod \"maas-api-revoke-sub-c2f65010-2rf7k\" (UID: \"4f7a55ca-e2b1-469e-b5fa-22d4558de0fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2f65010-2rf7k" Nov 03 20:55:34.689145 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:34.688976 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f7a55ca-e2b1-469e-b5fa-22d4558de0fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2f65010-2rf7k\" (UID: \"4f7a55ca-e2b1-469e-b5fa-22d4558de0fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2f65010-2rf7k" Nov 03 20:55:34.689145 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:34.689038 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f7a55ca-e2b1-469e-b5fa-22d4558de0fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2f65010-2rf7k\" (UID: \"4f7a55ca-e2b1-469e-b5fa-22d4558de0fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2f65010-2rf7k" Nov 03 20:55:34.689666 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:34.689650 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f7a55ca-e2b1-469e-b5fa-22d4558de0fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2f65010-2rf7k\" (UID: \"4f7a55ca-e2b1-469e-b5fa-22d4558de0fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2f65010-2rf7k" Nov 03 20:55:34.691498 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:34.691473 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f7a55ca-e2b1-469e-b5fa-22d4558de0fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2f65010-2rf7k\" (UID: \"4f7a55ca-e2b1-469e-b5fa-22d4558de0fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2f65010-2rf7k" Nov 03 20:55:34.749943 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:34.749915 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jvdl\" (UniqueName: \"kubernetes.io/projected/4f7a55ca-e2b1-469e-b5fa-22d4558de0fb-kube-api-access-2jvdl\") pod \"maas-api-revoke-sub-c2f65010-2rf7k\" (UID: \"4f7a55ca-e2b1-469e-b5fa-22d4558de0fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2f65010-2rf7k" Nov 03 20:55:34.818683 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:34.818599 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2f65010-2rf7k" Nov 03 20:55:34.888837 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:34.888795 4305 generic.go:356] "Generic (PLEG): container finished" podID="b6b6b70b-9fc3-4786-bf43-afedba0e31a3" containerID="3828920ba73658dcc620bca4fae8d2dabb522f981dc05db247e94cd1ea233791" exitCode=0 Nov 03 20:55:34.897619 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:34.897587 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a58fa67a-s5r5h" event={"ID":"b6b6b70b-9fc3-4786-bf43-afedba0e31a3","Type":"ContainerDied","Data":"3828920ba73658dcc620bca4fae8d2dabb522f981dc05db247e94cd1ea233791"} Nov 03 20:55:35.208103 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:35.208078 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2f65010-2rf7k"] Nov 03 20:55:35.209220 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:55:35.209193 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f7a55ca_e2b1_469e_b5fa_22d4558de0fb.slice/crio-df17b276fd268ea36d9e00f8397ac92a4d6c767c8973a1cb06ab7dff92155a98 WatchSource:0}: Error finding container df17b276fd268ea36d9e00f8397ac92a4d6c767c8973a1cb06ab7dff92155a98: Status 404 returned error can't find the container with id df17b276fd268ea36d9e00f8397ac92a4d6c767c8973a1cb06ab7dff92155a98 Nov 03 20:55:35.896145 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:35.896116 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2f65010-2rf7k" event={"ID":"4f7a55ca-e2b1-469e-b5fa-22d4558de0fb","Type":"ContainerStarted","Data":"7e428e941394b9fbdf1802809df5237008d3f74dc5c299a0ffe26dc185bd093d"} Nov 03 20:55:35.896586 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:35.896166 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2f65010-2rf7k" event={"ID":"4f7a55ca-e2b1-469e-b5fa-22d4558de0fb","Type":"ContainerStarted","Data":"df17b276fd268ea36d9e00f8397ac92a4d6c767c8973a1cb06ab7dff92155a98"} Nov 03 20:55:36.013130 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:36.013073 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2f65010-2rf7k" podStartSLOduration=2.013058127 podStartE2EDuration="2.013058127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:55:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:55:36.011149201 +0000 UTC m=+971.236468986" watchObservedRunningTime="2026-11-03 20:55:36.013058127 +0000 UTC m=+971.238377954" Nov 03 20:55:36.130201 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:36.130173 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a58fa67a-s5r5h" Nov 03 20:55:36.210168 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:36.210084 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6b6b70b-9fc3-4786-bf43-afedba0e31a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6b6b70b-9fc3-4786-bf43-afedba0e31a3-maas-api-xks-ca\") pod \"b6b6b70b-9fc3-4786-bf43-afedba0e31a3\" (UID: \"b6b6b70b-9fc3-4786-bf43-afedba0e31a3\") " Nov 03 20:55:36.210319 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:36.210181 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6b6b70b-9fc3-4786-bf43-afedba0e31a3-kube-api-access-t9nmr\" (UniqueName: \"kubernetes.io/projected/b6b6b70b-9fc3-4786-bf43-afedba0e31a3-kube-api-access-t9nmr\") pod \"b6b6b70b-9fc3-4786-bf43-afedba0e31a3\" (UID: \"b6b6b70b-9fc3-4786-bf43-afedba0e31a3\") " Nov 03 20:55:36.210319 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:36.210259 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6b6b70b-9fc3-4786-bf43-afedba0e31a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6b6b70b-9fc3-4786-bf43-afedba0e31a3-maas-api-openshift-ca\") pod \"b6b6b70b-9fc3-4786-bf43-afedba0e31a3\" (UID: \"b6b6b70b-9fc3-4786-bf43-afedba0e31a3\") " Nov 03 20:55:36.210616 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:36.210594 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6b6b70b-9fc3-4786-bf43-afedba0e31a3-maas-api-openshift-ca" pod "b6b6b70b-9fc3-4786-bf43-afedba0e31a3" (UID: "b6b6b70b-9fc3-4786-bf43-afedba0e31a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:55:36.212358 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:36.212334 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6b6b70b-9fc3-4786-bf43-afedba0e31a3-maas-api-xks-ca" pod "b6b6b70b-9fc3-4786-bf43-afedba0e31a3" (UID: "b6b6b70b-9fc3-4786-bf43-afedba0e31a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:55:36.212640 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:36.212614 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6b6b70b-9fc3-4786-bf43-afedba0e31a3-kube-api-access-t9nmr" pod "b6b6b70b-9fc3-4786-bf43-afedba0e31a3" (UID: "b6b6b70b-9fc3-4786-bf43-afedba0e31a3"). InnerVolumeSpecName "kube-api-access-t9nmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:55:36.311556 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:36.311518 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6b6b70b-9fc3-4786-bf43-afedba0e31a3-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:55:36.311556 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:36.311550 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6b6b70b-9fc3-4786-bf43-afedba0e31a3-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:55:36.311765 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:36.311564 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9nmr\" (UniqueName: \"kubernetes.io/projected/b6b6b70b-9fc3-4786-bf43-afedba0e31a3-kube-api-access-t9nmr\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:55:36.902664 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:36.902637 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a58fa67a-s5r5h" Nov 03 20:55:36.903169 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:36.902663 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a58fa67a-s5r5h" event={"ID":"b6b6b70b-9fc3-4786-bf43-afedba0e31a3","Type":"ContainerDied","Data":"c6435dc1e0b42971a061c7f70a8cf8d332a6fdf46dc95f417a58f5e52929b702"} Nov 03 20:55:36.903169 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:36.902798 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6435dc1e0b42971a061c7f70a8cf8d332a6fdf46dc95f417a58f5e52929b702" Nov 03 20:55:36.904920 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:36.904897 4305 generic.go:356] "Generic (PLEG): container finished" podID="4f7a55ca-e2b1-469e-b5fa-22d4558de0fb" containerID="7e428e941394b9fbdf1802809df5237008d3f74dc5c299a0ffe26dc185bd093d" exitCode=0 Nov 03 20:55:36.905024 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:36.904978 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2f65010-2rf7k" event={"ID":"4f7a55ca-e2b1-469e-b5fa-22d4558de0fb","Type":"ContainerDied","Data":"7e428e941394b9fbdf1802809df5237008d3f74dc5c299a0ffe26dc185bd093d"} Nov 03 20:55:38.065455 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:38.065409 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2f65010-2rf7k" Nov 03 20:55:38.134062 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:38.134028 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f7a55ca-e2b1-469e-b5fa-22d4558de0fb-kube-api-access-2jvdl\" (UniqueName: \"kubernetes.io/projected/4f7a55ca-e2b1-469e-b5fa-22d4558de0fb-kube-api-access-2jvdl\") pod \"4f7a55ca-e2b1-469e-b5fa-22d4558de0fb\" (UID: \"4f7a55ca-e2b1-469e-b5fa-22d4558de0fb\") " Nov 03 20:55:38.134218 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:38.134163 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f7a55ca-e2b1-469e-b5fa-22d4558de0fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f7a55ca-e2b1-469e-b5fa-22d4558de0fb-maas-api-xks-ca\") pod \"4f7a55ca-e2b1-469e-b5fa-22d4558de0fb\" (UID: \"4f7a55ca-e2b1-469e-b5fa-22d4558de0fb\") " Nov 03 20:55:38.134218 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:38.134214 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f7a55ca-e2b1-469e-b5fa-22d4558de0fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f7a55ca-e2b1-469e-b5fa-22d4558de0fb-maas-api-openshift-ca\") pod \"4f7a55ca-e2b1-469e-b5fa-22d4558de0fb\" (UID: \"4f7a55ca-e2b1-469e-b5fa-22d4558de0fb\") " Nov 03 20:55:38.134679 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:38.134645 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f7a55ca-e2b1-469e-b5fa-22d4558de0fb-maas-api-openshift-ca" pod "4f7a55ca-e2b1-469e-b5fa-22d4558de0fb" (UID: "4f7a55ca-e2b1-469e-b5fa-22d4558de0fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:55:38.136397 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:38.136374 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f7a55ca-e2b1-469e-b5fa-22d4558de0fb-maas-api-xks-ca" pod "4f7a55ca-e2b1-469e-b5fa-22d4558de0fb" (UID: "4f7a55ca-e2b1-469e-b5fa-22d4558de0fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:55:38.136496 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:38.136395 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f7a55ca-e2b1-469e-b5fa-22d4558de0fb-kube-api-access-2jvdl" pod "4f7a55ca-e2b1-469e-b5fa-22d4558de0fb" (UID: "4f7a55ca-e2b1-469e-b5fa-22d4558de0fb"). InnerVolumeSpecName "kube-api-access-2jvdl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:55:38.235638 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:38.235554 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f7a55ca-e2b1-469e-b5fa-22d4558de0fb-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:55:38.235638 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:38.235584 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f7a55ca-e2b1-469e-b5fa-22d4558de0fb-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:55:38.235638 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:38.235597 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jvdl\" (UniqueName: \"kubernetes.io/projected/4f7a55ca-e2b1-469e-b5fa-22d4558de0fb-kube-api-access-2jvdl\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:55:38.918326 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:38.918295 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2f65010-2rf7k" Nov 03 20:55:38.918495 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:38.918331 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2f65010-2rf7k" event={"ID":"4f7a55ca-e2b1-469e-b5fa-22d4558de0fb","Type":"ContainerDied","Data":"df17b276fd268ea36d9e00f8397ac92a4d6c767c8973a1cb06ab7dff92155a98"} Nov 03 20:55:38.918495 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:38.918380 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df17b276fd268ea36d9e00f8397ac92a4d6c767c8973a1cb06ab7dff92155a98" Nov 03 20:55:39.436060 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.435990 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jccsf" Nov 03 20:55:39.450823 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.450790 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc5b2941-a16d-4ab5-a36b-12a7f381c8c0-kube-api-access-ng4f6\" (UniqueName: \"kubernetes.io/projected/bc5b2941-a16d-4ab5-a36b-12a7f381c8c0-kube-api-access-ng4f6\") pod \"bc5b2941-a16d-4ab5-a36b-12a7f381c8c0\" (UID: \"bc5b2941-a16d-4ab5-a36b-12a7f381c8c0\") " Nov 03 20:55:39.451008 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.450908 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc5b2941-a16d-4ab5-a36b-12a7f381c8c0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc5b2941-a16d-4ab5-a36b-12a7f381c8c0-workload-certs\") pod \"bc5b2941-a16d-4ab5-a36b-12a7f381c8c0\" (UID: \"bc5b2941-a16d-4ab5-a36b-12a7f381c8c0\") " Nov 03 20:55:39.451008 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.450951 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc5b2941-a16d-4ab5-a36b-12a7f381c8c0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc5b2941-a16d-4ab5-a36b-12a7f381c8c0-workload-socket\") pod \"bc5b2941-a16d-4ab5-a36b-12a7f381c8c0\" (UID: \"bc5b2941-a16d-4ab5-a36b-12a7f381c8c0\") " Nov 03 20:55:39.451008 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.451001 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc5b2941-a16d-4ab5-a36b-12a7f381c8c0-istio-token\" (UniqueName: \"kubernetes.io/projected/bc5b2941-a16d-4ab5-a36b-12a7f381c8c0-istio-token\") pod \"bc5b2941-a16d-4ab5-a36b-12a7f381c8c0\" (UID: \"bc5b2941-a16d-4ab5-a36b-12a7f381c8c0\") " Nov 03 20:55:39.451183 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.451055 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc5b2941-a16d-4ab5-a36b-12a7f381c8c0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc5b2941-a16d-4ab5-a36b-12a7f381c8c0-credential-socket\") pod \"bc5b2941-a16d-4ab5-a36b-12a7f381c8c0\" (UID: \"bc5b2941-a16d-4ab5-a36b-12a7f381c8c0\") " Nov 03 20:55:39.451183 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.451119 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bc5b2941-a16d-4ab5-a36b-12a7f381c8c0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc5b2941-a16d-4ab5-a36b-12a7f381c8c0-istio-podinfo\") pod \"bc5b2941-a16d-4ab5-a36b-12a7f381c8c0\" (UID: \"bc5b2941-a16d-4ab5-a36b-12a7f381c8c0\") " Nov 03 20:55:39.451183 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.451162 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc5b2941-a16d-4ab5-a36b-12a7f381c8c0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc5b2941-a16d-4ab5-a36b-12a7f381c8c0-istio-data\") pod \"bc5b2941-a16d-4ab5-a36b-12a7f381c8c0\" (UID: \"bc5b2941-a16d-4ab5-a36b-12a7f381c8c0\") " Nov 03 20:55:39.451337 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.451192 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc5b2941-a16d-4ab5-a36b-12a7f381c8c0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc5b2941-a16d-4ab5-a36b-12a7f381c8c0-istiod-ca-cert\") pod \"bc5b2941-a16d-4ab5-a36b-12a7f381c8c0\" (UID: \"bc5b2941-a16d-4ab5-a36b-12a7f381c8c0\") " Nov 03 20:55:39.451337 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.451259 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc5b2941-a16d-4ab5-a36b-12a7f381c8c0-workload-certs" pod "bc5b2941-a16d-4ab5-a36b-12a7f381c8c0" (UID: "bc5b2941-a16d-4ab5-a36b-12a7f381c8c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:55:39.451337 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.451270 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc5b2941-a16d-4ab5-a36b-12a7f381c8c0-workload-socket" pod "bc5b2941-a16d-4ab5-a36b-12a7f381c8c0" (UID: "bc5b2941-a16d-4ab5-a36b-12a7f381c8c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:55:39.451337 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.451309 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc5b2941-a16d-4ab5-a36b-12a7f381c8c0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc5b2941-a16d-4ab5-a36b-12a7f381c8c0-istio-envoy\") pod \"bc5b2941-a16d-4ab5-a36b-12a7f381c8c0\" (UID: \"bc5b2941-a16d-4ab5-a36b-12a7f381c8c0\") " Nov 03 20:55:39.451751 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.451463 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc5b2941-a16d-4ab5-a36b-12a7f381c8c0-istio-data" pod "bc5b2941-a16d-4ab5-a36b-12a7f381c8c0" (UID: "bc5b2941-a16d-4ab5-a36b-12a7f381c8c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:55:39.451751 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.451693 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc5b2941-a16d-4ab5-a36b-12a7f381c8c0-istiod-ca-cert" pod "bc5b2941-a16d-4ab5-a36b-12a7f381c8c0" (UID: "bc5b2941-a16d-4ab5-a36b-12a7f381c8c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:55:39.451905 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.451554 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc5b2941-a16d-4ab5-a36b-12a7f381c8c0-credential-socket" pod "bc5b2941-a16d-4ab5-a36b-12a7f381c8c0" (UID: "bc5b2941-a16d-4ab5-a36b-12a7f381c8c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:55:39.451974 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.451869 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc5b2941-a16d-4ab5-a36b-12a7f381c8c0-workload-certs\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:55:39.451974 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.451942 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc5b2941-a16d-4ab5-a36b-12a7f381c8c0-workload-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:55:39.451974 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.451961 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc5b2941-a16d-4ab5-a36b-12a7f381c8c0-istio-data\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:55:39.452139 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.451975 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc5b2941-a16d-4ab5-a36b-12a7f381c8c0-istiod-ca-cert\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:55:39.453612 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.453586 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc5b2941-a16d-4ab5-a36b-12a7f381c8c0-kube-api-access-ng4f6" pod "bc5b2941-a16d-4ab5-a36b-12a7f381c8c0" (UID: "bc5b2941-a16d-4ab5-a36b-12a7f381c8c0"). InnerVolumeSpecName "kube-api-access-ng4f6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:55:39.453741 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.453701 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc5b2941-a16d-4ab5-a36b-12a7f381c8c0-istio-envoy" pod "bc5b2941-a16d-4ab5-a36b-12a7f381c8c0" (UID: "bc5b2941-a16d-4ab5-a36b-12a7f381c8c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:55:39.454269 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.454249 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bc5b2941-a16d-4ab5-a36b-12a7f381c8c0-istio-podinfo" pod "bc5b2941-a16d-4ab5-a36b-12a7f381c8c0" (UID: "bc5b2941-a16d-4ab5-a36b-12a7f381c8c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:55:39.454365 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.454267 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc5b2941-a16d-4ab5-a36b-12a7f381c8c0-istio-token" pod "bc5b2941-a16d-4ab5-a36b-12a7f381c8c0" (UID: "bc5b2941-a16d-4ab5-a36b-12a7f381c8c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:55:39.474950 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.474927 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7" Nov 03 20:55:39.552951 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.552918 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39ce2073-d106-4204-be5d-9ea93f510db8-istio-token\" (UniqueName: \"kubernetes.io/projected/39ce2073-d106-4204-be5d-9ea93f510db8-istio-token\") pod \"39ce2073-d106-4204-be5d-9ea93f510db8\" (UID: \"39ce2073-d106-4204-be5d-9ea93f510db8\") " Nov 03 20:55:39.552951 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.552958 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39ce2073-d106-4204-be5d-9ea93f510db8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39ce2073-d106-4204-be5d-9ea93f510db8-istiod-ca-cert\") pod \"39ce2073-d106-4204-be5d-9ea93f510db8\" (UID: \"39ce2073-d106-4204-be5d-9ea93f510db8\") " Nov 03 20:55:39.553179 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.553046 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/39ce2073-d106-4204-be5d-9ea93f510db8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39ce2073-d106-4204-be5d-9ea93f510db8-istio-podinfo\") pod \"39ce2073-d106-4204-be5d-9ea93f510db8\" (UID: \"39ce2073-d106-4204-be5d-9ea93f510db8\") " Nov 03 20:55:39.553179 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.553093 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39ce2073-d106-4204-be5d-9ea93f510db8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39ce2073-d106-4204-be5d-9ea93f510db8-credential-socket\") pod \"39ce2073-d106-4204-be5d-9ea93f510db8\" (UID: \"39ce2073-d106-4204-be5d-9ea93f510db8\") " Nov 03 20:55:39.553179 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.553122 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39ce2073-d106-4204-be5d-9ea93f510db8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39ce2073-d106-4204-be5d-9ea93f510db8-workload-socket\") pod \"39ce2073-d106-4204-be5d-9ea93f510db8\" (UID: \"39ce2073-d106-4204-be5d-9ea93f510db8\") " Nov 03 20:55:39.553321 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.553192 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39ce2073-d106-4204-be5d-9ea93f510db8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39ce2073-d106-4204-be5d-9ea93f510db8-istio-data\") pod \"39ce2073-d106-4204-be5d-9ea93f510db8\" (UID: \"39ce2073-d106-4204-be5d-9ea93f510db8\") " Nov 03 20:55:39.553321 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.553303 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39ce2073-d106-4204-be5d-9ea93f510db8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39ce2073-d106-4204-be5d-9ea93f510db8-istio-envoy\") pod \"39ce2073-d106-4204-be5d-9ea93f510db8\" (UID: \"39ce2073-d106-4204-be5d-9ea93f510db8\") " Nov 03 20:55:39.553422 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.553338 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39ce2073-d106-4204-be5d-9ea93f510db8-kube-api-access-2gqcs\" (UniqueName: \"kubernetes.io/projected/39ce2073-d106-4204-be5d-9ea93f510db8-kube-api-access-2gqcs\") pod \"39ce2073-d106-4204-be5d-9ea93f510db8\" (UID: \"39ce2073-d106-4204-be5d-9ea93f510db8\") " Nov 03 20:55:39.553422 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.553353 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39ce2073-d106-4204-be5d-9ea93f510db8-credential-socket" pod "39ce2073-d106-4204-be5d-9ea93f510db8" (UID: "39ce2073-d106-4204-be5d-9ea93f510db8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:55:39.553549 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.553461 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39ce2073-d106-4204-be5d-9ea93f510db8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39ce2073-d106-4204-be5d-9ea93f510db8-workload-certs\") pod \"39ce2073-d106-4204-be5d-9ea93f510db8\" (UID: \"39ce2073-d106-4204-be5d-9ea93f510db8\") " Nov 03 20:55:39.553549 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.553507 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39ce2073-d106-4204-be5d-9ea93f510db8-istiod-ca-cert" pod "39ce2073-d106-4204-be5d-9ea93f510db8" (UID: "39ce2073-d106-4204-be5d-9ea93f510db8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:55:39.553549 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.553510 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39ce2073-d106-4204-be5d-9ea93f510db8-workload-socket" pod "39ce2073-d106-4204-be5d-9ea93f510db8" (UID: "39ce2073-d106-4204-be5d-9ea93f510db8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:55:39.553698 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.553640 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39ce2073-d106-4204-be5d-9ea93f510db8-istio-data" pod "39ce2073-d106-4204-be5d-9ea93f510db8" (UID: "39ce2073-d106-4204-be5d-9ea93f510db8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:55:39.553805 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.553785 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39ce2073-d106-4204-be5d-9ea93f510db8-workload-certs" pod "39ce2073-d106-4204-be5d-9ea93f510db8" (UID: "39ce2073-d106-4204-be5d-9ea93f510db8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:55:39.554218 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.554193 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc5b2941-a16d-4ab5-a36b-12a7f381c8c0-istio-token\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:55:39.554345 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.554221 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc5b2941-a16d-4ab5-a36b-12a7f381c8c0-credential-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:55:39.554345 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.554236 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39ce2073-d106-4204-be5d-9ea93f510db8-workload-certs\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:55:39.554345 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.554250 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc5b2941-a16d-4ab5-a36b-12a7f381c8c0-istio-podinfo\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:55:39.554345 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.554266 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39ce2073-d106-4204-be5d-9ea93f510db8-istiod-ca-cert\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:55:39.554345 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.554280 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc5b2941-a16d-4ab5-a36b-12a7f381c8c0-istio-envoy\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:55:39.554345 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.554295 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39ce2073-d106-4204-be5d-9ea93f510db8-credential-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:55:39.554345 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.554311 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39ce2073-d106-4204-be5d-9ea93f510db8-workload-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:55:39.554345 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.554325 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ng4f6\" (UniqueName: \"kubernetes.io/projected/bc5b2941-a16d-4ab5-a36b-12a7f381c8c0-kube-api-access-ng4f6\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:55:39.554345 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.554338 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39ce2073-d106-4204-be5d-9ea93f510db8-istio-data\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:55:39.555466 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.555424 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39ce2073-d106-4204-be5d-9ea93f510db8-istio-token" pod "39ce2073-d106-4204-be5d-9ea93f510db8" (UID: "39ce2073-d106-4204-be5d-9ea93f510db8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:55:39.555630 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.555611 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39ce2073-d106-4204-be5d-9ea93f510db8-istio-podinfo" pod "39ce2073-d106-4204-be5d-9ea93f510db8" (UID: "39ce2073-d106-4204-be5d-9ea93f510db8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:55:39.555705 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.555629 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39ce2073-d106-4204-be5d-9ea93f510db8-kube-api-access-2gqcs" pod "39ce2073-d106-4204-be5d-9ea93f510db8" (UID: "39ce2073-d106-4204-be5d-9ea93f510db8"). InnerVolumeSpecName "kube-api-access-2gqcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:55:39.555705 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.555693 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39ce2073-d106-4204-be5d-9ea93f510db8-istio-envoy" pod "39ce2073-d106-4204-be5d-9ea93f510db8" (UID: "39ce2073-d106-4204-be5d-9ea93f510db8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:55:39.655564 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.655530 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39ce2073-d106-4204-be5d-9ea93f510db8-istio-token\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:55:39.655564 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.655561 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39ce2073-d106-4204-be5d-9ea93f510db8-istio-podinfo\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:55:39.655564 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.655570 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39ce2073-d106-4204-be5d-9ea93f510db8-istio-envoy\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:55:39.655801 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.655578 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gqcs\" (UniqueName: \"kubernetes.io/projected/39ce2073-d106-4204-be5d-9ea93f510db8-kube-api-access-2gqcs\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:55:39.926526 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.926491 4305 generic.go:356] "Generic (PLEG): container finished" podID="bc5b2941-a16d-4ab5-a36b-12a7f381c8c0" containerID="a7c6681dd4637d18897540b6f35b4bc5d27fdab63018f303004341edd08f427b" exitCode=0 Nov 03 20:55:39.926843 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.926529 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jccsf" event={"ID":"bc5b2941-a16d-4ab5-a36b-12a7f381c8c0","Type":"ContainerDied","Data":"a7c6681dd4637d18897540b6f35b4bc5d27fdab63018f303004341edd08f427b"} Nov 03 20:55:39.926843 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.926598 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jccsf" Nov 03 20:55:39.926843 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.926612 4305 scope.go:122] "RemoveContainer" containerID="a7c6681dd4637d18897540b6f35b4bc5d27fdab63018f303004341edd08f427b" Nov 03 20:55:39.926843 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.926599 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jccsf" event={"ID":"bc5b2941-a16d-4ab5-a36b-12a7f381c8c0","Type":"ContainerDied","Data":"b89fba9c5e754f8f718b5f3991377fe1322522ff87753e4de7189f116f3fd15c"} Nov 03 20:55:39.928532 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.928504 4305 generic.go:356] "Generic (PLEG): container finished" podID="39ce2073-d106-4204-be5d-9ea93f510db8" containerID="95f0c87e9c27adfd0113034e263a9cdd1a09ae70aee578bb2728fe2769424624" exitCode=0 Nov 03 20:55:39.928662 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.928543 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7" event={"ID":"39ce2073-d106-4204-be5d-9ea93f510db8","Type":"ContainerDied","Data":"95f0c87e9c27adfd0113034e263a9cdd1a09ae70aee578bb2728fe2769424624"} Nov 03 20:55:39.928662 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.928585 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7" Nov 03 20:55:39.928662 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.928614 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7" event={"ID":"39ce2073-d106-4204-be5d-9ea93f510db8","Type":"ContainerDied","Data":"521fd60f5cdf8a032a8fd0e3aeed9fb7dee62b1cbd7f6610b378de2e51d9d36c"} Nov 03 20:55:39.938413 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.938373 4305 scope.go:122] "RemoveContainer" containerID="a7c6681dd4637d18897540b6f35b4bc5d27fdab63018f303004341edd08f427b" Nov 03 20:55:39.938906 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:55:39.938772 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7c6681dd4637d18897540b6f35b4bc5d27fdab63018f303004341edd08f427b\": container with ID starting with a7c6681dd4637d18897540b6f35b4bc5d27fdab63018f303004341edd08f427b not found: ID does not exist" containerID="a7c6681dd4637d18897540b6f35b4bc5d27fdab63018f303004341edd08f427b" Nov 03 20:55:39.938906 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.938808 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7c6681dd4637d18897540b6f35b4bc5d27fdab63018f303004341edd08f427b"} err="failed to get container status \"a7c6681dd4637d18897540b6f35b4bc5d27fdab63018f303004341edd08f427b\": rpc error: code = NotFound desc = could not find container \"a7c6681dd4637d18897540b6f35b4bc5d27fdab63018f303004341edd08f427b\": container with ID starting with a7c6681dd4637d18897540b6f35b4bc5d27fdab63018f303004341edd08f427b not found: ID does not exist" Nov 03 20:55:39.938906 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.938833 4305 scope.go:122] "RemoveContainer" containerID="95f0c87e9c27adfd0113034e263a9cdd1a09ae70aee578bb2728fe2769424624" Nov 03 20:55:39.948512 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.948491 4305 scope.go:122] "RemoveContainer" containerID="95f0c87e9c27adfd0113034e263a9cdd1a09ae70aee578bb2728fe2769424624" Nov 03 20:55:39.948795 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:55:39.948774 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95f0c87e9c27adfd0113034e263a9cdd1a09ae70aee578bb2728fe2769424624\": container with ID starting with 95f0c87e9c27adfd0113034e263a9cdd1a09ae70aee578bb2728fe2769424624 not found: ID does not exist" containerID="95f0c87e9c27adfd0113034e263a9cdd1a09ae70aee578bb2728fe2769424624" Nov 03 20:55:39.948871 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:39.948801 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95f0c87e9c27adfd0113034e263a9cdd1a09ae70aee578bb2728fe2769424624"} err="failed to get container status \"95f0c87e9c27adfd0113034e263a9cdd1a09ae70aee578bb2728fe2769424624\": rpc error: code = NotFound desc = could not find container \"95f0c87e9c27adfd0113034e263a9cdd1a09ae70aee578bb2728fe2769424624\": container with ID starting with 95f0c87e9c27adfd0113034e263a9cdd1a09ae70aee578bb2728fe2769424624 not found: ID does not exist" Nov 03 20:55:40.092792 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:40.092756 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7"] Nov 03 20:55:40.096992 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:40.096948 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lxqg7"] Nov 03 20:55:40.206565 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:40.206531 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jccsf"] Nov 03 20:55:40.209324 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:40.209299 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jccsf"] Nov 03 20:55:40.896226 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:40.896197 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39ce2073-d106-4204-be5d-9ea93f510db8" path="/var/lib/kubelet/pods/39ce2073-d106-4204-be5d-9ea93f510db8/volumes" Nov 03 20:55:40.896699 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:40.896684 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc5b2941-a16d-4ab5-a36b-12a7f381c8c0" path="/var/lib/kubelet/pods/bc5b2941-a16d-4ab5-a36b-12a7f381c8c0/volumes" Nov 03 20:55:46.321854 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:55:46.321817 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 20:55:56.256714 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:56.256680 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-659eab4c-drdgf"] Nov 03 20:55:56.257218 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:56.257205 4305 cpu_manager.go:414] "containerMap: removing container" podUID="bc5b2941-a16d-4ab5-a36b-12a7f381c8c0" containerName="istio-proxy" Nov 03 20:55:56.257261 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:56.257218 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc5b2941-a16d-4ab5-a36b-12a7f381c8c0" containerName="istio-proxy" Nov 03 20:55:56.257261 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:56.257228 4305 cpu_manager.go:414] "containerMap: removing container" podUID="4f7a55ca-e2b1-469e-b5fa-22d4558de0fb" containerName="revoke-keys" Nov 03 20:55:56.257261 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:56.257233 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f7a55ca-e2b1-469e-b5fa-22d4558de0fb" containerName="revoke-keys" Nov 03 20:55:56.257261 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:56.257240 4305 cpu_manager.go:414] "containerMap: removing container" podUID="39ce2073-d106-4204-be5d-9ea93f510db8" containerName="istio-proxy" Nov 03 20:55:56.257261 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:56.257245 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39ce2073-d106-4204-be5d-9ea93f510db8" containerName="istio-proxy" Nov 03 20:55:56.257261 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:56.257252 4305 cpu_manager.go:414] "containerMap: removing container" podUID="b6b6b70b-9fc3-4786-bf43-afedba0e31a3" containerName="revoke-keys" Nov 03 20:55:56.257261 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:56.257256 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6b6b70b-9fc3-4786-bf43-afedba0e31a3" containerName="revoke-keys" Nov 03 20:55:56.257483 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:56.257339 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6b6b70b-9fc3-4786-bf43-afedba0e31a3" containerName="revoke-keys" Nov 03 20:55:56.257483 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:56.257347 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc5b2941-a16d-4ab5-a36b-12a7f381c8c0" containerName="istio-proxy" Nov 03 20:55:56.257483 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:56.257353 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f7a55ca-e2b1-469e-b5fa-22d4558de0fb" containerName="revoke-keys" Nov 03 20:55:56.257483 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:56.257358 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="39ce2073-d106-4204-be5d-9ea93f510db8" containerName="istio-proxy" Nov 03 20:55:56.261904 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:56.261884 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-659eab4c-drdgf" Nov 03 20:55:56.315337 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:56.315309 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:55:56.364747 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:56.364715 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77vtd\"" Nov 03 20:55:56.364747 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:56.364715 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:55:56.364962 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:56.364715 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:55:56.367479 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:56.367456 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r59hd\" (UniqueName: \"kubernetes.io/projected/48077494-2ea6-4d8f-aab2-32afc6872ae1-kube-api-access-r59hd\") pod \"maas-api-revoke-sub-659eab4c-drdgf\" (UID: \"48077494-2ea6-4d8f-aab2-32afc6872ae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-659eab4c-drdgf" Nov 03 20:55:56.367701 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:56.367682 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48077494-2ea6-4d8f-aab2-32afc6872ae1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-659eab4c-drdgf\" (UID: \"48077494-2ea6-4d8f-aab2-32afc6872ae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-659eab4c-drdgf" Nov 03 20:55:56.367744 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:56.367714 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48077494-2ea6-4d8f-aab2-32afc6872ae1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-659eab4c-drdgf\" (UID: \"48077494-2ea6-4d8f-aab2-32afc6872ae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-659eab4c-drdgf" Nov 03 20:55:56.367853 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:56.367828 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-659eab4c-drdgf"] Nov 03 20:55:56.469817 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:56.469787 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48077494-2ea6-4d8f-aab2-32afc6872ae1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-659eab4c-drdgf\" (UID: \"48077494-2ea6-4d8f-aab2-32afc6872ae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-659eab4c-drdgf" Nov 03 20:55:56.469817 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:56.469823 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48077494-2ea6-4d8f-aab2-32afc6872ae1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-659eab4c-drdgf\" (UID: \"48077494-2ea6-4d8f-aab2-32afc6872ae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-659eab4c-drdgf" Nov 03 20:55:56.470036 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:56.469955 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r59hd\" (UniqueName: \"kubernetes.io/projected/48077494-2ea6-4d8f-aab2-32afc6872ae1-kube-api-access-r59hd\") pod \"maas-api-revoke-sub-659eab4c-drdgf\" (UID: \"48077494-2ea6-4d8f-aab2-32afc6872ae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-659eab4c-drdgf" Nov 03 20:55:56.470384 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:56.470359 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48077494-2ea6-4d8f-aab2-32afc6872ae1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-659eab4c-drdgf\" (UID: \"48077494-2ea6-4d8f-aab2-32afc6872ae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-659eab4c-drdgf" Nov 03 20:55:56.472316 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:56.472295 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48077494-2ea6-4d8f-aab2-32afc6872ae1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-659eab4c-drdgf\" (UID: \"48077494-2ea6-4d8f-aab2-32afc6872ae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-659eab4c-drdgf" Nov 03 20:55:56.531201 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:56.531131 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r59hd\" (UniqueName: \"kubernetes.io/projected/48077494-2ea6-4d8f-aab2-32afc6872ae1-kube-api-access-r59hd\") pod \"maas-api-revoke-sub-659eab4c-drdgf\" (UID: \"48077494-2ea6-4d8f-aab2-32afc6872ae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-659eab4c-drdgf" Nov 03 20:55:56.576196 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:56.576168 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-659eab4c-drdgf" Nov 03 20:55:56.952726 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:56.952695 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-659eab4c-drdgf"] Nov 03 20:55:56.953675 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:55:56.953651 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48077494_2ea6_4d8f_aab2_32afc6872ae1.slice/crio-748cdd3fab4c6c79b69e4de43a21d103081707e711871917a98f1dcee1ce62ca WatchSource:0}: Error finding container 748cdd3fab4c6c79b69e4de43a21d103081707e711871917a98f1dcee1ce62ca: Status 404 returned error can't find the container with id 748cdd3fab4c6c79b69e4de43a21d103081707e711871917a98f1dcee1ce62ca Nov 03 20:55:57.033747 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:57.033713 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-659eab4c-drdgf" event={"ID":"48077494-2ea6-4d8f-aab2-32afc6872ae1","Type":"ContainerStarted","Data":"748cdd3fab4c6c79b69e4de43a21d103081707e711871917a98f1dcee1ce62ca"} Nov 03 20:55:58.040967 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:58.040935 4305 generic.go:356] "Generic (PLEG): container finished" podID="48077494-2ea6-4d8f-aab2-32afc6872ae1" containerID="3611f92a517f15afb6c596e185871b41ab8061bab24a9ad3399537f241abf346" exitCode=0 Nov 03 20:55:58.041387 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:58.041032 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-659eab4c-drdgf" event={"ID":"48077494-2ea6-4d8f-aab2-32afc6872ae1","Type":"ContainerDied","Data":"3611f92a517f15afb6c596e185871b41ab8061bab24a9ad3399537f241abf346"} Nov 03 20:55:59.197938 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:59.197914 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-659eab4c-drdgf" Nov 03 20:55:59.303376 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:59.303296 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48077494-2ea6-4d8f-aab2-32afc6872ae1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48077494-2ea6-4d8f-aab2-32afc6872ae1-maas-api-openshift-ca\") pod \"48077494-2ea6-4d8f-aab2-32afc6872ae1\" (UID: \"48077494-2ea6-4d8f-aab2-32afc6872ae1\") " Nov 03 20:55:59.303555 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:59.303425 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48077494-2ea6-4d8f-aab2-32afc6872ae1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48077494-2ea6-4d8f-aab2-32afc6872ae1-maas-api-xks-ca\") pod \"48077494-2ea6-4d8f-aab2-32afc6872ae1\" (UID: \"48077494-2ea6-4d8f-aab2-32afc6872ae1\") " Nov 03 20:55:59.303555 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:59.303529 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48077494-2ea6-4d8f-aab2-32afc6872ae1-kube-api-access-r59hd\" (UniqueName: \"kubernetes.io/projected/48077494-2ea6-4d8f-aab2-32afc6872ae1-kube-api-access-r59hd\") pod \"48077494-2ea6-4d8f-aab2-32afc6872ae1\" (UID: \"48077494-2ea6-4d8f-aab2-32afc6872ae1\") " Nov 03 20:55:59.303714 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:59.303694 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48077494-2ea6-4d8f-aab2-32afc6872ae1-maas-api-openshift-ca" pod "48077494-2ea6-4d8f-aab2-32afc6872ae1" (UID: "48077494-2ea6-4d8f-aab2-32afc6872ae1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:55:59.303996 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:59.303971 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48077494-2ea6-4d8f-aab2-32afc6872ae1-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:55:59.305721 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:59.305699 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48077494-2ea6-4d8f-aab2-32afc6872ae1-maas-api-xks-ca" pod "48077494-2ea6-4d8f-aab2-32afc6872ae1" (UID: "48077494-2ea6-4d8f-aab2-32afc6872ae1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:55:59.305804 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:59.305741 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48077494-2ea6-4d8f-aab2-32afc6872ae1-kube-api-access-r59hd" pod "48077494-2ea6-4d8f-aab2-32afc6872ae1" (UID: "48077494-2ea6-4d8f-aab2-32afc6872ae1"). InnerVolumeSpecName "kube-api-access-r59hd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:55:59.405407 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:59.405376 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48077494-2ea6-4d8f-aab2-32afc6872ae1-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:55:59.405407 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:55:59.405403 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r59hd\" (UniqueName: \"kubernetes.io/projected/48077494-2ea6-4d8f-aab2-32afc6872ae1-kube-api-access-r59hd\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:56:00.053857 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:00.053808 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-659eab4c-drdgf" event={"ID":"48077494-2ea6-4d8f-aab2-32afc6872ae1","Type":"ContainerDied","Data":"748cdd3fab4c6c79b69e4de43a21d103081707e711871917a98f1dcee1ce62ca"} Nov 03 20:56:00.053857 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:00.053863 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="748cdd3fab4c6c79b69e4de43a21d103081707e711871917a98f1dcee1ce62ca" Nov 03 20:56:00.054047 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:00.053825 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-659eab4c-drdgf" Nov 03 20:56:03.494835 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:03.494801 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fca8d49-xqrvf"] Nov 03 20:56:03.495311 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:03.495298 4305 cpu_manager.go:414] "containerMap: removing container" podUID="48077494-2ea6-4d8f-aab2-32afc6872ae1" containerName="revoke-keys" Nov 03 20:56:03.495349 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:03.495312 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48077494-2ea6-4d8f-aab2-32afc6872ae1" containerName="revoke-keys" Nov 03 20:56:03.495428 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:03.495419 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="48077494-2ea6-4d8f-aab2-32afc6872ae1" containerName="revoke-keys" Nov 03 20:56:03.498891 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:03.498874 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fca8d49-xqrvf" Nov 03 20:56:03.556832 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:03.556799 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fca8d49-xqrvf"] Nov 03 20:56:03.602672 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:03.602637 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:56:03.602672 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:03.602665 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:56:03.602926 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:03.602671 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:56:03.602926 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:03.602696 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77vtd\"" Nov 03 20:56:03.654006 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:03.653966 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a15184d3-975f-4101-9823-3686d63d3f8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fca8d49-xqrvf\" (UID: \"a15184d3-975f-4101-9823-3686d63d3f8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fca8d49-xqrvf" Nov 03 20:56:03.654213 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:03.654058 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq9x9\" (UniqueName: \"kubernetes.io/projected/a15184d3-975f-4101-9823-3686d63d3f8d-kube-api-access-tq9x9\") pod \"maas-api-revoke-sub-7fca8d49-xqrvf\" (UID: \"a15184d3-975f-4101-9823-3686d63d3f8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fca8d49-xqrvf" Nov 03 20:56:03.654280 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:03.654247 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a15184d3-975f-4101-9823-3686d63d3f8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fca8d49-xqrvf\" (UID: \"a15184d3-975f-4101-9823-3686d63d3f8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fca8d49-xqrvf" Nov 03 20:56:03.755160 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:03.755079 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a15184d3-975f-4101-9823-3686d63d3f8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fca8d49-xqrvf\" (UID: \"a15184d3-975f-4101-9823-3686d63d3f8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fca8d49-xqrvf" Nov 03 20:56:03.755316 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:03.755183 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a15184d3-975f-4101-9823-3686d63d3f8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fca8d49-xqrvf\" (UID: \"a15184d3-975f-4101-9823-3686d63d3f8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fca8d49-xqrvf" Nov 03 20:56:03.755316 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:03.755216 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq9x9\" (UniqueName: \"kubernetes.io/projected/a15184d3-975f-4101-9823-3686d63d3f8d-kube-api-access-tq9x9\") pod \"maas-api-revoke-sub-7fca8d49-xqrvf\" (UID: \"a15184d3-975f-4101-9823-3686d63d3f8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fca8d49-xqrvf" Nov 03 20:56:03.755741 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:03.755721 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a15184d3-975f-4101-9823-3686d63d3f8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fca8d49-xqrvf\" (UID: \"a15184d3-975f-4101-9823-3686d63d3f8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fca8d49-xqrvf" Nov 03 20:56:03.757729 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:03.757705 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a15184d3-975f-4101-9823-3686d63d3f8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fca8d49-xqrvf\" (UID: \"a15184d3-975f-4101-9823-3686d63d3f8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fca8d49-xqrvf" Nov 03 20:56:03.816091 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:03.816053 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq9x9\" (UniqueName: \"kubernetes.io/projected/a15184d3-975f-4101-9823-3686d63d3f8d-kube-api-access-tq9x9\") pod \"maas-api-revoke-sub-7fca8d49-xqrvf\" (UID: \"a15184d3-975f-4101-9823-3686d63d3f8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fca8d49-xqrvf" Nov 03 20:56:04.113066 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:04.112954 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fca8d49-xqrvf" Nov 03 20:56:04.375736 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:04.375661 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 20:56:04.486990 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:04.486966 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fca8d49-xqrvf"] Nov 03 20:56:04.489201 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:56:04.489172 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda15184d3_975f_4101_9823_3686d63d3f8d.slice/crio-80bd7daefb95461b9c16db1a21cb224cec2640dc459d26bf16a0914a2cecbf9f WatchSource:0}: Error finding container 80bd7daefb95461b9c16db1a21cb224cec2640dc459d26bf16a0914a2cecbf9f: Status 404 returned error can't find the container with id 80bd7daefb95461b9c16db1a21cb224cec2640dc459d26bf16a0914a2cecbf9f Nov 03 20:56:05.083283 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:05.083195 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fca8d49-xqrvf" event={"ID":"a15184d3-975f-4101-9823-3686d63d3f8d","Type":"ContainerStarted","Data":"398a0ff7840008cd05ecead6644e9fb12c6161160a0629c6ae6a4a4dce152680"} Nov 03 20:56:05.083283 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:05.083246 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fca8d49-xqrvf" event={"ID":"a15184d3-975f-4101-9823-3686d63d3f8d","Type":"ContainerStarted","Data":"80bd7daefb95461b9c16db1a21cb224cec2640dc459d26bf16a0914a2cecbf9f"} Nov 03 20:56:05.203598 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:05.203551 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fca8d49-xqrvf" podStartSLOduration=2.203537441 podStartE2EDuration="2.203537441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:56:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:56:05.200169685 +0000 UTC m=+1000.425489478" watchObservedRunningTime="2026-11-03 20:56:05.203537441 +0000 UTC m=+1000.428857257" Nov 03 20:56:06.093282 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:06.093239 4305 generic.go:356] "Generic (PLEG): container finished" podID="a15184d3-975f-4101-9823-3686d63d3f8d" containerID="398a0ff7840008cd05ecead6644e9fb12c6161160a0629c6ae6a4a4dce152680" exitCode=0 Nov 03 20:56:06.093748 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:06.093332 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fca8d49-xqrvf" event={"ID":"a15184d3-975f-4101-9823-3686d63d3f8d","Type":"ContainerDied","Data":"398a0ff7840008cd05ecead6644e9fb12c6161160a0629c6ae6a4a4dce152680"} Nov 03 20:56:07.014573 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:07.014539 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-812c2c2c-h24x7"] Nov 03 20:56:07.018659 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:07.018634 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-812c2c2c-h24x7" Nov 03 20:56:07.078636 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:07.078598 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-812c2c2c-h24x7"] Nov 03 20:56:07.093685 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:07.093647 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b3bcecb-831c-49d0-a4ee-72c965fe9872-maas-api-xks-ca\") pod \"maas-api-revoke-sub-812c2c2c-h24x7\" (UID: \"1b3bcecb-831c-49d0-a4ee-72c965fe9872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-812c2c2c-h24x7" Nov 03 20:56:07.094221 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:07.093709 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b3bcecb-831c-49d0-a4ee-72c965fe9872-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-812c2c2c-h24x7\" (UID: \"1b3bcecb-831c-49d0-a4ee-72c965fe9872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-812c2c2c-h24x7" Nov 03 20:56:07.094221 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:07.093744 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rdbm\" (UniqueName: \"kubernetes.io/projected/1b3bcecb-831c-49d0-a4ee-72c965fe9872-kube-api-access-8rdbm\") pod \"maas-api-revoke-sub-812c2c2c-h24x7\" (UID: \"1b3bcecb-831c-49d0-a4ee-72c965fe9872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-812c2c2c-h24x7" Nov 03 20:56:07.196597 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:07.195553 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b3bcecb-831c-49d0-a4ee-72c965fe9872-maas-api-xks-ca\") pod \"maas-api-revoke-sub-812c2c2c-h24x7\" (UID: \"1b3bcecb-831c-49d0-a4ee-72c965fe9872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-812c2c2c-h24x7" Nov 03 20:56:07.196597 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:07.195616 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b3bcecb-831c-49d0-a4ee-72c965fe9872-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-812c2c2c-h24x7\" (UID: \"1b3bcecb-831c-49d0-a4ee-72c965fe9872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-812c2c2c-h24x7" Nov 03 20:56:07.196597 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:07.195647 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rdbm\" (UniqueName: \"kubernetes.io/projected/1b3bcecb-831c-49d0-a4ee-72c965fe9872-kube-api-access-8rdbm\") pod \"maas-api-revoke-sub-812c2c2c-h24x7\" (UID: \"1b3bcecb-831c-49d0-a4ee-72c965fe9872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-812c2c2c-h24x7" Nov 03 20:56:07.197007 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:07.196967 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b3bcecb-831c-49d0-a4ee-72c965fe9872-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-812c2c2c-h24x7\" (UID: \"1b3bcecb-831c-49d0-a4ee-72c965fe9872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-812c2c2c-h24x7" Nov 03 20:56:07.203153 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:07.203089 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b3bcecb-831c-49d0-a4ee-72c965fe9872-maas-api-xks-ca\") pod \"maas-api-revoke-sub-812c2c2c-h24x7\" (UID: \"1b3bcecb-831c-49d0-a4ee-72c965fe9872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-812c2c2c-h24x7" Nov 03 20:56:07.257215 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:07.257188 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rdbm\" (UniqueName: \"kubernetes.io/projected/1b3bcecb-831c-49d0-a4ee-72c965fe9872-kube-api-access-8rdbm\") pod \"maas-api-revoke-sub-812c2c2c-h24x7\" (UID: \"1b3bcecb-831c-49d0-a4ee-72c965fe9872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-812c2c2c-h24x7" Nov 03 20:56:07.260282 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:07.260263 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fca8d49-xqrvf" Nov 03 20:56:07.332544 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:07.332462 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-812c2c2c-h24x7" Nov 03 20:56:07.396967 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:07.396939 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a15184d3-975f-4101-9823-3686d63d3f8d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a15184d3-975f-4101-9823-3686d63d3f8d-maas-api-xks-ca\") pod \"a15184d3-975f-4101-9823-3686d63d3f8d\" (UID: \"a15184d3-975f-4101-9823-3686d63d3f8d\") " Nov 03 20:56:07.397598 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:07.397304 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a15184d3-975f-4101-9823-3686d63d3f8d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a15184d3-975f-4101-9823-3686d63d3f8d-maas-api-openshift-ca\") pod \"a15184d3-975f-4101-9823-3686d63d3f8d\" (UID: \"a15184d3-975f-4101-9823-3686d63d3f8d\") " Nov 03 20:56:07.397786 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:07.397766 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a15184d3-975f-4101-9823-3686d63d3f8d-kube-api-access-tq9x9\" (UniqueName: \"kubernetes.io/projected/a15184d3-975f-4101-9823-3686d63d3f8d-kube-api-access-tq9x9\") pod \"a15184d3-975f-4101-9823-3686d63d3f8d\" (UID: \"a15184d3-975f-4101-9823-3686d63d3f8d\") " Nov 03 20:56:07.398879 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:07.398824 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a15184d3-975f-4101-9823-3686d63d3f8d-maas-api-openshift-ca" pod "a15184d3-975f-4101-9823-3686d63d3f8d" (UID: "a15184d3-975f-4101-9823-3686d63d3f8d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:56:07.400019 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:07.399996 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a15184d3-975f-4101-9823-3686d63d3f8d-maas-api-xks-ca" pod "a15184d3-975f-4101-9823-3686d63d3f8d" (UID: "a15184d3-975f-4101-9823-3686d63d3f8d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:56:07.401190 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:07.401166 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a15184d3-975f-4101-9823-3686d63d3f8d-kube-api-access-tq9x9" pod "a15184d3-975f-4101-9823-3686d63d3f8d" (UID: "a15184d3-975f-4101-9823-3686d63d3f8d"). InnerVolumeSpecName "kube-api-access-tq9x9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:56:07.499729 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:07.499695 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a15184d3-975f-4101-9823-3686d63d3f8d-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:56:07.499729 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:07.499724 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a15184d3-975f-4101-9823-3686d63d3f8d-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:56:07.499975 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:07.499743 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq9x9\" (UniqueName: \"kubernetes.io/projected/a15184d3-975f-4101-9823-3686d63d3f8d-kube-api-access-tq9x9\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:56:07.714377 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:07.714346 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-812c2c2c-h24x7"] Nov 03 20:56:07.716224 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:56:07.716197 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b3bcecb_831c_49d0_a4ee_72c965fe9872.slice/crio-a330fbe4961d486c11c095c9aa41341cd2b0da743c3e982d4524a0ef1cceb281 WatchSource:0}: Error finding container a330fbe4961d486c11c095c9aa41341cd2b0da743c3e982d4524a0ef1cceb281: Status 404 returned error can't find the container with id a330fbe4961d486c11c095c9aa41341cd2b0da743c3e982d4524a0ef1cceb281 Nov 03 20:56:08.108190 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:08.108164 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fca8d49-xqrvf" Nov 03 20:56:08.108190 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:08.108173 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fca8d49-xqrvf" event={"ID":"a15184d3-975f-4101-9823-3686d63d3f8d","Type":"ContainerDied","Data":"80bd7daefb95461b9c16db1a21cb224cec2640dc459d26bf16a0914a2cecbf9f"} Nov 03 20:56:08.108713 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:08.108232 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80bd7daefb95461b9c16db1a21cb224cec2640dc459d26bf16a0914a2cecbf9f" Nov 03 20:56:08.109925 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:08.109895 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-812c2c2c-h24x7" event={"ID":"1b3bcecb-831c-49d0-a4ee-72c965fe9872","Type":"ContainerStarted","Data":"0b5f22ea14aa96b3eaf56c9f3ec7a14cf95cfef50b7c58a4ccb6a8735e9fad96"} Nov 03 20:56:08.110047 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:08.109939 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-812c2c2c-h24x7" event={"ID":"1b3bcecb-831c-49d0-a4ee-72c965fe9872","Type":"ContainerStarted","Data":"a330fbe4961d486c11c095c9aa41341cd2b0da743c3e982d4524a0ef1cceb281"} Nov 03 20:56:08.226226 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:08.226128 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-812c2c2c-h24x7" podStartSLOduration=2.226113168 podStartE2EDuration="2.226113168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:56:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:56:08.22485552 +0000 UTC m=+1003.450175353" watchObservedRunningTime="2026-11-03 20:56:08.226113168 +0000 UTC m=+1003.451432963" Nov 03 20:56:09.119138 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:09.119056 4305 generic.go:356] "Generic (PLEG): container finished" podID="1b3bcecb-831c-49d0-a4ee-72c965fe9872" containerID="0b5f22ea14aa96b3eaf56c9f3ec7a14cf95cfef50b7c58a4ccb6a8735e9fad96" exitCode=0 Nov 03 20:56:09.119625 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:09.119139 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-812c2c2c-h24x7" event={"ID":"1b3bcecb-831c-49d0-a4ee-72c965fe9872","Type":"ContainerDied","Data":"0b5f22ea14aa96b3eaf56c9f3ec7a14cf95cfef50b7c58a4ccb6a8735e9fad96"} Nov 03 20:56:10.280704 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:10.280676 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-812c2c2c-h24x7" Nov 03 20:56:10.432311 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:10.432267 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b3bcecb-831c-49d0-a4ee-72c965fe9872-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b3bcecb-831c-49d0-a4ee-72c965fe9872-maas-api-xks-ca\") pod \"1b3bcecb-831c-49d0-a4ee-72c965fe9872\" (UID: \"1b3bcecb-831c-49d0-a4ee-72c965fe9872\") " Nov 03 20:56:10.432311 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:10.432308 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b3bcecb-831c-49d0-a4ee-72c965fe9872-kube-api-access-8rdbm\" (UniqueName: \"kubernetes.io/projected/1b3bcecb-831c-49d0-a4ee-72c965fe9872-kube-api-access-8rdbm\") pod \"1b3bcecb-831c-49d0-a4ee-72c965fe9872\" (UID: \"1b3bcecb-831c-49d0-a4ee-72c965fe9872\") " Nov 03 20:56:10.432636 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:10.432356 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b3bcecb-831c-49d0-a4ee-72c965fe9872-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b3bcecb-831c-49d0-a4ee-72c965fe9872-maas-api-openshift-ca\") pod \"1b3bcecb-831c-49d0-a4ee-72c965fe9872\" (UID: \"1b3bcecb-831c-49d0-a4ee-72c965fe9872\") " Nov 03 20:56:10.432843 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:10.432815 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b3bcecb-831c-49d0-a4ee-72c965fe9872-maas-api-openshift-ca" pod "1b3bcecb-831c-49d0-a4ee-72c965fe9872" (UID: "1b3bcecb-831c-49d0-a4ee-72c965fe9872"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:56:10.432999 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:10.432960 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b3bcecb-831c-49d0-a4ee-72c965fe9872-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:56:10.434675 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:10.434648 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b3bcecb-831c-49d0-a4ee-72c965fe9872-maas-api-xks-ca" pod "1b3bcecb-831c-49d0-a4ee-72c965fe9872" (UID: "1b3bcecb-831c-49d0-a4ee-72c965fe9872"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:56:10.435225 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:10.435207 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b3bcecb-831c-49d0-a4ee-72c965fe9872-kube-api-access-8rdbm" pod "1b3bcecb-831c-49d0-a4ee-72c965fe9872" (UID: "1b3bcecb-831c-49d0-a4ee-72c965fe9872"). InnerVolumeSpecName "kube-api-access-8rdbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:56:10.533893 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:10.533856 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b3bcecb-831c-49d0-a4ee-72c965fe9872-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:56:10.533893 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:10.533884 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rdbm\" (UniqueName: \"kubernetes.io/projected/1b3bcecb-831c-49d0-a4ee-72c965fe9872-kube-api-access-8rdbm\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:56:11.133618 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:11.133587 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-812c2c2c-h24x7" Nov 03 20:56:11.133818 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:11.133619 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-812c2c2c-h24x7" event={"ID":"1b3bcecb-831c-49d0-a4ee-72c965fe9872","Type":"ContainerDied","Data":"a330fbe4961d486c11c095c9aa41341cd2b0da743c3e982d4524a0ef1cceb281"} Nov 03 20:56:11.133818 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:11.133665 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a330fbe4961d486c11c095c9aa41341cd2b0da743c3e982d4524a0ef1cceb281" Nov 03 20:56:11.613414 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:11.613380 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0550170-xdbmz"] Nov 03 20:56:11.613917 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:11.613902 4305 cpu_manager.go:414] "containerMap: removing container" podUID="a15184d3-975f-4101-9823-3686d63d3f8d" containerName="revoke-keys" Nov 03 20:56:11.613958 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:11.613918 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a15184d3-975f-4101-9823-3686d63d3f8d" containerName="revoke-keys" Nov 03 20:56:11.613958 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:11.613933 4305 cpu_manager.go:414] "containerMap: removing container" podUID="1b3bcecb-831c-49d0-a4ee-72c965fe9872" containerName="revoke-keys" Nov 03 20:56:11.613958 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:11.613938 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b3bcecb-831c-49d0-a4ee-72c965fe9872" containerName="revoke-keys" Nov 03 20:56:11.614060 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:11.614050 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="a15184d3-975f-4101-9823-3686d63d3f8d" containerName="revoke-keys" Nov 03 20:56:11.614096 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:11.614060 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b3bcecb-831c-49d0-a4ee-72c965fe9872" containerName="revoke-keys" Nov 03 20:56:11.617209 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:11.617193 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0550170-xdbmz" Nov 03 20:56:11.670555 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:11.670511 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:56:11.670555 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:11.670569 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:56:11.720236 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:11.720198 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:56:11.720457 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:11.720338 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77vtd\"" Nov 03 20:56:11.723207 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:11.723185 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0550170-xdbmz"] Nov 03 20:56:11.748103 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:11.748075 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bdd446d-01b5-40a3-869d-0e7199d7a64f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0550170-xdbmz\" (UID: \"4bdd446d-01b5-40a3-869d-0e7199d7a64f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0550170-xdbmz" Nov 03 20:56:11.748236 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:11.748112 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bdd446d-01b5-40a3-869d-0e7199d7a64f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0550170-xdbmz\" (UID: \"4bdd446d-01b5-40a3-869d-0e7199d7a64f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0550170-xdbmz" Nov 03 20:56:11.748236 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:11.748139 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxljv\" (UniqueName: \"kubernetes.io/projected/4bdd446d-01b5-40a3-869d-0e7199d7a64f-kube-api-access-rxljv\") pod \"maas-api-revoke-sub-b0550170-xdbmz\" (UID: \"4bdd446d-01b5-40a3-869d-0e7199d7a64f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0550170-xdbmz" Nov 03 20:56:11.849200 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:11.849150 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bdd446d-01b5-40a3-869d-0e7199d7a64f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0550170-xdbmz\" (UID: \"4bdd446d-01b5-40a3-869d-0e7199d7a64f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0550170-xdbmz" Nov 03 20:56:11.849404 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:11.849204 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bdd446d-01b5-40a3-869d-0e7199d7a64f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0550170-xdbmz\" (UID: \"4bdd446d-01b5-40a3-869d-0e7199d7a64f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0550170-xdbmz" Nov 03 20:56:11.849404 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:11.849245 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxljv\" (UniqueName: \"kubernetes.io/projected/4bdd446d-01b5-40a3-869d-0e7199d7a64f-kube-api-access-rxljv\") pod \"maas-api-revoke-sub-b0550170-xdbmz\" (UID: \"4bdd446d-01b5-40a3-869d-0e7199d7a64f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0550170-xdbmz" Nov 03 20:56:11.849859 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:11.849840 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bdd446d-01b5-40a3-869d-0e7199d7a64f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0550170-xdbmz\" (UID: \"4bdd446d-01b5-40a3-869d-0e7199d7a64f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0550170-xdbmz" Nov 03 20:56:11.851673 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:11.851657 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bdd446d-01b5-40a3-869d-0e7199d7a64f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0550170-xdbmz\" (UID: \"4bdd446d-01b5-40a3-869d-0e7199d7a64f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0550170-xdbmz" Nov 03 20:56:11.910275 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:11.910239 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxljv\" (UniqueName: \"kubernetes.io/projected/4bdd446d-01b5-40a3-869d-0e7199d7a64f-kube-api-access-rxljv\") pod \"maas-api-revoke-sub-b0550170-xdbmz\" (UID: \"4bdd446d-01b5-40a3-869d-0e7199d7a64f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0550170-xdbmz" Nov 03 20:56:11.930733 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:11.930705 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0550170-xdbmz" Nov 03 20:56:12.314220 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:12.314101 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0550170-xdbmz"] Nov 03 20:56:12.315536 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:56:12.315503 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bdd446d_01b5_40a3_869d_0e7199d7a64f.slice/crio-e3a4860a74647c9af2cd9a4c11154c9005b5d4a6df6ea1efc526ca10d51cbf06 WatchSource:0}: Error finding container e3a4860a74647c9af2cd9a4c11154c9005b5d4a6df6ea1efc526ca10d51cbf06: Status 404 returned error can't find the container with id e3a4860a74647c9af2cd9a4c11154c9005b5d4a6df6ea1efc526ca10d51cbf06 Nov 03 20:56:13.147348 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:13.147313 4305 generic.go:356] "Generic (PLEG): container finished" podID="4bdd446d-01b5-40a3-869d-0e7199d7a64f" containerID="589020f0d49141260b3eb465465fe871c6429598454b16571789e29ea4426d89" exitCode=0 Nov 03 20:56:13.147786 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:13.147393 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0550170-xdbmz" event={"ID":"4bdd446d-01b5-40a3-869d-0e7199d7a64f","Type":"ContainerDied","Data":"589020f0d49141260b3eb465465fe871c6429598454b16571789e29ea4426d89"} Nov 03 20:56:13.147786 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:13.147474 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0550170-xdbmz" event={"ID":"4bdd446d-01b5-40a3-869d-0e7199d7a64f","Type":"ContainerStarted","Data":"e3a4860a74647c9af2cd9a4c11154c9005b5d4a6df6ea1efc526ca10d51cbf06"} Nov 03 20:56:14.314165 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:14.314136 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0550170-xdbmz" Nov 03 20:56:14.482048 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:14.481963 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bdd446d-01b5-40a3-869d-0e7199d7a64f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bdd446d-01b5-40a3-869d-0e7199d7a64f-maas-api-xks-ca\") pod \"4bdd446d-01b5-40a3-869d-0e7199d7a64f\" (UID: \"4bdd446d-01b5-40a3-869d-0e7199d7a64f\") " Nov 03 20:56:14.482211 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:14.482068 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bdd446d-01b5-40a3-869d-0e7199d7a64f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bdd446d-01b5-40a3-869d-0e7199d7a64f-maas-api-openshift-ca\") pod \"4bdd446d-01b5-40a3-869d-0e7199d7a64f\" (UID: \"4bdd446d-01b5-40a3-869d-0e7199d7a64f\") " Nov 03 20:56:14.482211 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:14.482111 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bdd446d-01b5-40a3-869d-0e7199d7a64f-kube-api-access-rxljv\" (UniqueName: \"kubernetes.io/projected/4bdd446d-01b5-40a3-869d-0e7199d7a64f-kube-api-access-rxljv\") pod \"4bdd446d-01b5-40a3-869d-0e7199d7a64f\" (UID: \"4bdd446d-01b5-40a3-869d-0e7199d7a64f\") " Nov 03 20:56:14.482538 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:14.482504 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bdd446d-01b5-40a3-869d-0e7199d7a64f-maas-api-openshift-ca" pod "4bdd446d-01b5-40a3-869d-0e7199d7a64f" (UID: "4bdd446d-01b5-40a3-869d-0e7199d7a64f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:56:14.482703 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:14.482689 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bdd446d-01b5-40a3-869d-0e7199d7a64f-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:56:14.484504 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:14.484418 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bdd446d-01b5-40a3-869d-0e7199d7a64f-maas-api-xks-ca" pod "4bdd446d-01b5-40a3-869d-0e7199d7a64f" (UID: "4bdd446d-01b5-40a3-869d-0e7199d7a64f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:56:14.484504 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:14.484424 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bdd446d-01b5-40a3-869d-0e7199d7a64f-kube-api-access-rxljv" pod "4bdd446d-01b5-40a3-869d-0e7199d7a64f" (UID: "4bdd446d-01b5-40a3-869d-0e7199d7a64f"). InnerVolumeSpecName "kube-api-access-rxljv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:56:14.583848 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:14.583808 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bdd446d-01b5-40a3-869d-0e7199d7a64f-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:56:14.583848 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:14.583835 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxljv\" (UniqueName: \"kubernetes.io/projected/4bdd446d-01b5-40a3-869d-0e7199d7a64f-kube-api-access-rxljv\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:56:15.162080 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:15.162002 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0550170-xdbmz" Nov 03 20:56:15.162080 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:15.162019 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0550170-xdbmz" event={"ID":"4bdd446d-01b5-40a3-869d-0e7199d7a64f","Type":"ContainerDied","Data":"e3a4860a74647c9af2cd9a4c11154c9005b5d4a6df6ea1efc526ca10d51cbf06"} Nov 03 20:56:15.162080 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:15.162058 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3a4860a74647c9af2cd9a4c11154c9005b5d4a6df6ea1efc526ca10d51cbf06" Nov 03 20:56:16.321986 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:56:16.321952 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 20:56:19.539889 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.539831 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-588hz"] Nov 03 20:56:19.540414 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.540400 4305 cpu_manager.go:414] "containerMap: removing container" podUID="4bdd446d-01b5-40a3-869d-0e7199d7a64f" containerName="revoke-keys" Nov 03 20:56:19.540539 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.540417 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bdd446d-01b5-40a3-869d-0e7199d7a64f" containerName="revoke-keys" Nov 03 20:56:19.540597 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.540550 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bdd446d-01b5-40a3-869d-0e7199d7a64f" containerName="revoke-keys" Nov 03 20:56:19.544287 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.544266 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-588hz" Nov 03 20:56:19.557603 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.557574 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-59cjd"] Nov 03 20:56:19.561602 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.561578 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-59cjd" Nov 03 20:56:19.605459 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.605406 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-588hz"] Nov 03 20:56:19.633490 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.633427 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-59cjd\" (UID: \"a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-59cjd" Nov 03 20:56:19.633684 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.633550 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnhpg\" (UniqueName: \"kubernetes.io/projected/327b41fd-c8a7-4926-bc59-8869ae87aa8b-kube-api-access-mnhpg\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-588hz\" (UID: \"327b41fd-c8a7-4926-bc59-8869ae87aa8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-588hz" Nov 03 20:56:19.633684 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.633575 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-59cjd\" (UID: \"a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-59cjd" Nov 03 20:56:19.633684 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.633662 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-59cjd\" (UID: \"a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-59cjd" Nov 03 20:56:19.633864 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.633694 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/327b41fd-c8a7-4926-bc59-8869ae87aa8b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-588hz\" (UID: \"327b41fd-c8a7-4926-bc59-8869ae87aa8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-588hz" Nov 03 20:56:19.633864 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.633727 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/327b41fd-c8a7-4926-bc59-8869ae87aa8b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-588hz\" (UID: \"327b41fd-c8a7-4926-bc59-8869ae87aa8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-588hz" Nov 03 20:56:19.633864 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.633829 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/327b41fd-c8a7-4926-bc59-8869ae87aa8b-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-588hz\" (UID: \"327b41fd-c8a7-4926-bc59-8869ae87aa8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-588hz" Nov 03 20:56:19.634013 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.633875 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnxwt\" (UniqueName: \"kubernetes.io/projected/a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a-kube-api-access-bnxwt\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-59cjd\" (UID: \"a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-59cjd" Nov 03 20:56:19.634013 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.634005 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/327b41fd-c8a7-4926-bc59-8869ae87aa8b-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-588hz\" (UID: \"327b41fd-c8a7-4926-bc59-8869ae87aa8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-588hz" Nov 03 20:56:19.634119 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.634054 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/327b41fd-c8a7-4926-bc59-8869ae87aa8b-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-588hz\" (UID: \"327b41fd-c8a7-4926-bc59-8869ae87aa8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-588hz" Nov 03 20:56:19.634172 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.634133 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/327b41fd-c8a7-4926-bc59-8869ae87aa8b-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-588hz\" (UID: \"327b41fd-c8a7-4926-bc59-8869ae87aa8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-588hz" Nov 03 20:56:19.634230 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.634204 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-59cjd\" (UID: \"a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-59cjd" Nov 03 20:56:19.634323 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.634306 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/327b41fd-c8a7-4926-bc59-8869ae87aa8b-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-588hz\" (UID: \"327b41fd-c8a7-4926-bc59-8869ae87aa8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-588hz" Nov 03 20:56:19.634387 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.634335 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-59cjd\" (UID: \"a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-59cjd" Nov 03 20:56:19.634460 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.634400 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-59cjd\" (UID: \"a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-59cjd" Nov 03 20:56:19.634524 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.634469 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-59cjd\" (UID: \"a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-59cjd" Nov 03 20:56:19.634524 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.634496 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/327b41fd-c8a7-4926-bc59-8869ae87aa8b-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-588hz\" (UID: \"327b41fd-c8a7-4926-bc59-8869ae87aa8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-588hz" Nov 03 20:56:19.634630 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.634520 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-59cjd\" (UID: \"a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-59cjd" Nov 03 20:56:19.645341 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.645305 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b319d281-57hgb"] Nov 03 20:56:19.649467 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.649411 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b319d281-57hgb" Nov 03 20:56:19.657068 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.657041 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-59cjd"] Nov 03 20:56:19.704685 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.704650 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:56:19.704685 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.704650 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:56:19.704977 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.704705 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77vtd\"" Nov 03 20:56:19.704977 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.704792 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:56:19.708776 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.708751 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b319d281-57hgb"] Nov 03 20:56:19.736223 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.736183 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/327b41fd-c8a7-4926-bc59-8869ae87aa8b-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-588hz\" (UID: \"327b41fd-c8a7-4926-bc59-8869ae87aa8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-588hz" Nov 03 20:56:19.736223 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.736224 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-59cjd\" (UID: \"a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-59cjd" Nov 03 20:56:19.736530 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.736255 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-59cjd\" (UID: \"a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-59cjd" Nov 03 20:56:19.736530 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.736281 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-59cjd\" (UID: \"a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-59cjd" Nov 03 20:56:19.736530 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.736308 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/327b41fd-c8a7-4926-bc59-8869ae87aa8b-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-588hz\" (UID: \"327b41fd-c8a7-4926-bc59-8869ae87aa8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-588hz" Nov 03 20:56:19.736530 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.736328 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-59cjd\" (UID: \"a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-59cjd" Nov 03 20:56:19.736530 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.736352 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-59cjd\" (UID: \"a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-59cjd" Nov 03 20:56:19.736530 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.736385 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnhpg\" (UniqueName: \"kubernetes.io/projected/327b41fd-c8a7-4926-bc59-8869ae87aa8b-kube-api-access-mnhpg\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-588hz\" (UID: \"327b41fd-c8a7-4926-bc59-8869ae87aa8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-588hz" Nov 03 20:56:19.736530 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.736409 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-59cjd\" (UID: \"a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-59cjd" Nov 03 20:56:19.736530 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.736485 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65e231f1-20dc-4c54-b7f1-4217ab1a0965-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b319d281-57hgb\" (UID: \"65e231f1-20dc-4c54-b7f1-4217ab1a0965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b319d281-57hgb" Nov 03 20:56:19.736530 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.736521 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-59cjd\" (UID: \"a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-59cjd" Nov 03 20:56:19.737007 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.736552 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/327b41fd-c8a7-4926-bc59-8869ae87aa8b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-588hz\" (UID: \"327b41fd-c8a7-4926-bc59-8869ae87aa8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-588hz" Nov 03 20:56:19.737007 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.736589 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/327b41fd-c8a7-4926-bc59-8869ae87aa8b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-588hz\" (UID: \"327b41fd-c8a7-4926-bc59-8869ae87aa8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-588hz" Nov 03 20:56:19.737007 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.736620 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/327b41fd-c8a7-4926-bc59-8869ae87aa8b-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-588hz\" (UID: \"327b41fd-c8a7-4926-bc59-8869ae87aa8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-588hz" Nov 03 20:56:19.737007 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.736658 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnxwt\" (UniqueName: \"kubernetes.io/projected/a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a-kube-api-access-bnxwt\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-59cjd\" (UID: \"a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-59cjd" Nov 03 20:56:19.737007 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.736686 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-59cjd\" (UID: \"a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-59cjd" Nov 03 20:56:19.737007 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.736729 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/327b41fd-c8a7-4926-bc59-8869ae87aa8b-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-588hz\" (UID: \"327b41fd-c8a7-4926-bc59-8869ae87aa8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-588hz" Nov 03 20:56:19.737007 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.736885 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/327b41fd-c8a7-4926-bc59-8869ae87aa8b-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-588hz\" (UID: \"327b41fd-c8a7-4926-bc59-8869ae87aa8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-588hz" Nov 03 20:56:19.737007 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.736957 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/327b41fd-c8a7-4926-bc59-8869ae87aa8b-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-588hz\" (UID: \"327b41fd-c8a7-4926-bc59-8869ae87aa8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-588hz" Nov 03 20:56:19.737007 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.737006 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65e231f1-20dc-4c54-b7f1-4217ab1a0965-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b319d281-57hgb\" (UID: \"65e231f1-20dc-4c54-b7f1-4217ab1a0965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b319d281-57hgb" Nov 03 20:56:19.737418 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.737038 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/327b41fd-c8a7-4926-bc59-8869ae87aa8b-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-588hz\" (UID: \"327b41fd-c8a7-4926-bc59-8869ae87aa8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-588hz" Nov 03 20:56:19.737418 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.737101 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8fjj\" (UniqueName: \"kubernetes.io/projected/65e231f1-20dc-4c54-b7f1-4217ab1a0965-kube-api-access-t8fjj\") pod \"maas-api-revoke-sub-b319d281-57hgb\" (UID: \"65e231f1-20dc-4c54-b7f1-4217ab1a0965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b319d281-57hgb" Nov 03 20:56:19.737418 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.737147 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-59cjd\" (UID: \"a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-59cjd" Nov 03 20:56:19.737609 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.737451 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/327b41fd-c8a7-4926-bc59-8869ae87aa8b-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-588hz\" (UID: \"327b41fd-c8a7-4926-bc59-8869ae87aa8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-588hz" Nov 03 20:56:19.737737 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.737714 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/327b41fd-c8a7-4926-bc59-8869ae87aa8b-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-588hz\" (UID: \"327b41fd-c8a7-4926-bc59-8869ae87aa8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-588hz" Nov 03 20:56:19.737942 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.737827 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/327b41fd-c8a7-4926-bc59-8869ae87aa8b-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-588hz\" (UID: \"327b41fd-c8a7-4926-bc59-8869ae87aa8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-588hz" Nov 03 20:56:19.737942 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.737895 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-59cjd\" (UID: \"a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-59cjd" Nov 03 20:56:19.738403 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.738119 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-59cjd\" (UID: \"a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-59cjd" Nov 03 20:56:19.738403 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.738127 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-59cjd\" (UID: \"a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-59cjd" Nov 03 20:56:19.738403 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.738258 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/327b41fd-c8a7-4926-bc59-8869ae87aa8b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-588hz\" (UID: \"327b41fd-c8a7-4926-bc59-8869ae87aa8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-588hz" Nov 03 20:56:19.738403 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.738366 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-59cjd\" (UID: \"a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-59cjd" Nov 03 20:56:19.740073 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.740050 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-59cjd\" (UID: \"a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-59cjd" Nov 03 20:56:19.740156 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.740087 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/327b41fd-c8a7-4926-bc59-8869ae87aa8b-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-588hz\" (UID: \"327b41fd-c8a7-4926-bc59-8869ae87aa8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-588hz" Nov 03 20:56:19.740392 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.740364 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/327b41fd-c8a7-4926-bc59-8869ae87aa8b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-588hz\" (UID: \"327b41fd-c8a7-4926-bc59-8869ae87aa8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-588hz" Nov 03 20:56:19.740960 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.740938 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-59cjd\" (UID: \"a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-59cjd" Nov 03 20:56:19.800197 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.800105 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-59cjd\" (UID: \"a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-59cjd" Nov 03 20:56:19.800364 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.800256 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnhpg\" (UniqueName: \"kubernetes.io/projected/327b41fd-c8a7-4926-bc59-8869ae87aa8b-kube-api-access-mnhpg\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-588hz\" (UID: \"327b41fd-c8a7-4926-bc59-8869ae87aa8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-588hz" Nov 03 20:56:19.800518 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.800462 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnxwt\" (UniqueName: \"kubernetes.io/projected/a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a-kube-api-access-bnxwt\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-59cjd\" (UID: \"a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-59cjd" Nov 03 20:56:19.809098 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.809039 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/327b41fd-c8a7-4926-bc59-8869ae87aa8b-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-588hz\" (UID: \"327b41fd-c8a7-4926-bc59-8869ae87aa8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-588hz" Nov 03 20:56:19.838376 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.838319 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65e231f1-20dc-4c54-b7f1-4217ab1a0965-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b319d281-57hgb\" (UID: \"65e231f1-20dc-4c54-b7f1-4217ab1a0965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b319d281-57hgb" Nov 03 20:56:19.838376 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.838376 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8fjj\" (UniqueName: \"kubernetes.io/projected/65e231f1-20dc-4c54-b7f1-4217ab1a0965-kube-api-access-t8fjj\") pod \"maas-api-revoke-sub-b319d281-57hgb\" (UID: \"65e231f1-20dc-4c54-b7f1-4217ab1a0965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b319d281-57hgb" Nov 03 20:56:19.838737 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.838516 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65e231f1-20dc-4c54-b7f1-4217ab1a0965-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b319d281-57hgb\" (UID: \"65e231f1-20dc-4c54-b7f1-4217ab1a0965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b319d281-57hgb" Nov 03 20:56:19.839170 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.839133 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65e231f1-20dc-4c54-b7f1-4217ab1a0965-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b319d281-57hgb\" (UID: \"65e231f1-20dc-4c54-b7f1-4217ab1a0965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b319d281-57hgb" Nov 03 20:56:19.840903 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.840872 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65e231f1-20dc-4c54-b7f1-4217ab1a0965-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b319d281-57hgb\" (UID: \"65e231f1-20dc-4c54-b7f1-4217ab1a0965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b319d281-57hgb" Nov 03 20:56:19.858159 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.858124 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-588hz" Nov 03 20:56:19.875944 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.875907 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-59cjd" Nov 03 20:56:19.905761 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.905725 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8fjj\" (UniqueName: \"kubernetes.io/projected/65e231f1-20dc-4c54-b7f1-4217ab1a0965-kube-api-access-t8fjj\") pod \"maas-api-revoke-sub-b319d281-57hgb\" (UID: \"65e231f1-20dc-4c54-b7f1-4217ab1a0965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b319d281-57hgb" Nov 03 20:56:19.976466 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:19.974009 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b319d281-57hgb" Nov 03 20:56:20.307726 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:20.306803 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-588hz"] Nov 03 20:56:20.311784 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:20.311689 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-59cjd"] Nov 03 20:56:20.315682 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:20.315637 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:56:20.315797 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:20.315691 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:56:20.315797 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:20.315723 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:56:20.315797 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:20.315748 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:56:20.315797 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:20.315766 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:56:20.315797 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:20.315788 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:56:20.415339 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:20.415292 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b319d281-57hgb"] Nov 03 20:56:21.285473 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:21.285225 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b319d281-57hgb" event={"ID":"65e231f1-20dc-4c54-b7f1-4217ab1a0965","Type":"ContainerStarted","Data":"b633fbda66c098af94074ddacecd8b846671760567a1e73b4855582b12453a23"} Nov 03 20:56:21.285473 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:21.285295 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b319d281-57hgb" event={"ID":"65e231f1-20dc-4c54-b7f1-4217ab1a0965","Type":"ContainerStarted","Data":"f531edcd873d032c46e9dba818a0be2bc31464866f2e2b8a8bf97887cedb0aec"} Nov 03 20:56:21.308462 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:21.305983 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9311f42a-6pdhs"] Nov 03 20:56:21.359459 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:21.358739 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9311f42a-6pdhs"] Nov 03 20:56:21.359459 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:21.358794 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-59cjd" event={"ID":"a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a","Type":"ContainerStarted","Data":"578e5a417d1a34d198a42091b4a70573ccb2ce9f030428a3cc36eb5db596531c"} Nov 03 20:56:21.359459 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:21.358831 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-59cjd" event={"ID":"a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a","Type":"ContainerStarted","Data":"04475958e0ce2c10410fa5191c8a8698eb2ad2788608f576faf940c67b1553e6"} Nov 03 20:56:21.359459 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:21.358855 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-588hz" event={"ID":"327b41fd-c8a7-4926-bc59-8869ae87aa8b","Type":"ContainerStarted","Data":"f423dd3aac35b5995843da8d11913875dfa9f8109624326e4f6739579de731d4"} Nov 03 20:56:21.359459 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:21.358884 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-588hz" event={"ID":"327b41fd-c8a7-4926-bc59-8869ae87aa8b","Type":"ContainerStarted","Data":"afc7433a032c1b3bb7d663e9dd30c69a90d0bf860a87e62affe141b1c9d352c0"} Nov 03 20:56:21.359459 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:21.358988 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9311f42a-6pdhs" Nov 03 20:56:21.536523 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:21.533664 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn8p4\" (UniqueName: \"kubernetes.io/projected/17a26a30-1c68-44c7-ac11-2269b4d2b3d4-kube-api-access-dn8p4\") pod \"maas-api-revoke-sub-9311f42a-6pdhs\" (UID: \"17a26a30-1c68-44c7-ac11-2269b4d2b3d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9311f42a-6pdhs" Nov 03 20:56:21.536523 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:21.533717 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17a26a30-1c68-44c7-ac11-2269b4d2b3d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9311f42a-6pdhs\" (UID: \"17a26a30-1c68-44c7-ac11-2269b4d2b3d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9311f42a-6pdhs" Nov 03 20:56:21.536523 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:21.533860 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17a26a30-1c68-44c7-ac11-2269b4d2b3d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9311f42a-6pdhs\" (UID: \"17a26a30-1c68-44c7-ac11-2269b4d2b3d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9311f42a-6pdhs" Nov 03 20:56:21.638898 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:21.592652 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b319d281-57hgb" podStartSLOduration=2.592628816 podStartE2EDuration="2.592628816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:56:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:56:21.454006117 +0000 UTC m=+1016.679325908" watchObservedRunningTime="2026-11-03 20:56:21.592628816 +0000 UTC m=+1016.817948609" Nov 03 20:56:21.638898 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:21.635137 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn8p4\" (UniqueName: \"kubernetes.io/projected/17a26a30-1c68-44c7-ac11-2269b4d2b3d4-kube-api-access-dn8p4\") pod \"maas-api-revoke-sub-9311f42a-6pdhs\" (UID: \"17a26a30-1c68-44c7-ac11-2269b4d2b3d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9311f42a-6pdhs" Nov 03 20:56:21.638898 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:21.635182 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17a26a30-1c68-44c7-ac11-2269b4d2b3d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9311f42a-6pdhs\" (UID: \"17a26a30-1c68-44c7-ac11-2269b4d2b3d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9311f42a-6pdhs" Nov 03 20:56:21.638898 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:21.635273 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17a26a30-1c68-44c7-ac11-2269b4d2b3d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9311f42a-6pdhs\" (UID: \"17a26a30-1c68-44c7-ac11-2269b4d2b3d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9311f42a-6pdhs" Nov 03 20:56:21.638898 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:21.636402 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17a26a30-1c68-44c7-ac11-2269b4d2b3d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9311f42a-6pdhs\" (UID: \"17a26a30-1c68-44c7-ac11-2269b4d2b3d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9311f42a-6pdhs" Nov 03 20:56:21.644853 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:21.644817 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17a26a30-1c68-44c7-ac11-2269b4d2b3d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9311f42a-6pdhs\" (UID: \"17a26a30-1c68-44c7-ac11-2269b4d2b3d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9311f42a-6pdhs" Nov 03 20:56:21.710617 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:21.710580 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn8p4\" (UniqueName: \"kubernetes.io/projected/17a26a30-1c68-44c7-ac11-2269b4d2b3d4-kube-api-access-dn8p4\") pod \"maas-api-revoke-sub-9311f42a-6pdhs\" (UID: \"17a26a30-1c68-44c7-ac11-2269b4d2b3d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9311f42a-6pdhs" Nov 03 20:56:21.804692 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:21.803638 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-588hz" podStartSLOduration=2.803615889 podStartE2EDuration="2.803615889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:56:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:56:21.775782259 +0000 UTC m=+1017.001102070" watchObservedRunningTime="2026-11-03 20:56:21.803615889 +0000 UTC m=+1017.028935680" Nov 03 20:56:21.866736 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:21.866700 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-588hz" Nov 03 20:56:21.867083 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:21.867063 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-588hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 20:56:21.867267 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:21.867242 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-588hz" podUID="327b41fd-c8a7-4926-bc59-8869ae87aa8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 20:56:21.887710 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:21.887675 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-59cjd" Nov 03 20:56:21.891502 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:21.891469 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-59cjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 20:56:21.891765 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:21.891740 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-59cjd" podUID="a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 20:56:21.903969 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:21.897743 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-59cjd" podStartSLOduration=2.897724042 podStartE2EDuration="2.897724042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:56:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:56:21.896102869 +0000 UTC m=+1017.121422717" watchObservedRunningTime="2026-11-03 20:56:21.897724042 +0000 UTC m=+1017.123043833" Nov 03 20:56:21.925114 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:21.925079 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9311f42a-6pdhs" Nov 03 20:56:22.333154 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:22.333117 4305 generic.go:356] "Generic (PLEG): container finished" podID="65e231f1-20dc-4c54-b7f1-4217ab1a0965" containerID="b633fbda66c098af94074ddacecd8b846671760567a1e73b4855582b12453a23" exitCode=0 Nov 03 20:56:22.333665 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:22.333201 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b319d281-57hgb" event={"ID":"65e231f1-20dc-4c54-b7f1-4217ab1a0965","Type":"ContainerDied","Data":"b633fbda66c098af94074ddacecd8b846671760567a1e73b4855582b12453a23"} Nov 03 20:56:22.344990 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:22.344969 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9311f42a-6pdhs"] Nov 03 20:56:22.345286 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:56:22.345267 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17a26a30_1c68_44c7_ac11_2269b4d2b3d4.slice/crio-36ccc5d4882d9339e78509c6a509ed7a85c207549ea7ef882c4bf956a87c8b71 WatchSource:0}: Error finding container 36ccc5d4882d9339e78509c6a509ed7a85c207549ea7ef882c4bf956a87c8b71: Status 404 returned error can't find the container with id 36ccc5d4882d9339e78509c6a509ed7a85c207549ea7ef882c4bf956a87c8b71 Nov 03 20:56:22.865425 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:22.865332 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-588hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 20:56:22.865744 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:22.865706 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-588hz" podUID="327b41fd-c8a7-4926-bc59-8869ae87aa8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 20:56:22.886375 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:22.886252 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-59cjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 20:56:22.886375 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:22.886316 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-59cjd" podUID="a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 20:56:23.119180 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:23.119096 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be167f9a-cldkf"] Nov 03 20:56:23.123062 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:23.123044 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be167f9a-cldkf" Nov 03 20:56:23.200888 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:23.200847 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be167f9a-cldkf"] Nov 03 20:56:23.295460 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:23.277748 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60a33d48-25bd-4b1c-91da-911e825718c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be167f9a-cldkf\" (UID: \"60a33d48-25bd-4b1c-91da-911e825718c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be167f9a-cldkf" Nov 03 20:56:23.296690 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:23.296655 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60a33d48-25bd-4b1c-91da-911e825718c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be167f9a-cldkf\" (UID: \"60a33d48-25bd-4b1c-91da-911e825718c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be167f9a-cldkf" Nov 03 20:56:23.297065 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:23.297045 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr98t\" (UniqueName: \"kubernetes.io/projected/60a33d48-25bd-4b1c-91da-911e825718c6-kube-api-access-fr98t\") pod \"maas-api-revoke-sub-be167f9a-cldkf\" (UID: \"60a33d48-25bd-4b1c-91da-911e825718c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be167f9a-cldkf" Nov 03 20:56:23.342353 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:23.342319 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9311f42a-6pdhs" event={"ID":"17a26a30-1c68-44c7-ac11-2269b4d2b3d4","Type":"ContainerStarted","Data":"d9814b638bf2433354ec201446b0e221cba4994f028d9c4a448b177d0622bb02"} Nov 03 20:56:23.342850 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:23.342379 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9311f42a-6pdhs" event={"ID":"17a26a30-1c68-44c7-ac11-2269b4d2b3d4","Type":"ContainerStarted","Data":"36ccc5d4882d9339e78509c6a509ed7a85c207549ea7ef882c4bf956a87c8b71"} Nov 03 20:56:23.400462 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:23.398075 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60a33d48-25bd-4b1c-91da-911e825718c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be167f9a-cldkf\" (UID: \"60a33d48-25bd-4b1c-91da-911e825718c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be167f9a-cldkf" Nov 03 20:56:23.400462 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:23.398156 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60a33d48-25bd-4b1c-91da-911e825718c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be167f9a-cldkf\" (UID: \"60a33d48-25bd-4b1c-91da-911e825718c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be167f9a-cldkf" Nov 03 20:56:23.400462 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:23.398254 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr98t\" (UniqueName: \"kubernetes.io/projected/60a33d48-25bd-4b1c-91da-911e825718c6-kube-api-access-fr98t\") pod \"maas-api-revoke-sub-be167f9a-cldkf\" (UID: \"60a33d48-25bd-4b1c-91da-911e825718c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be167f9a-cldkf" Nov 03 20:56:23.400462 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:23.399505 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60a33d48-25bd-4b1c-91da-911e825718c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be167f9a-cldkf\" (UID: \"60a33d48-25bd-4b1c-91da-911e825718c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be167f9a-cldkf" Nov 03 20:56:23.407607 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:23.407553 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60a33d48-25bd-4b1c-91da-911e825718c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be167f9a-cldkf\" (UID: \"60a33d48-25bd-4b1c-91da-911e825718c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be167f9a-cldkf" Nov 03 20:56:23.509170 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:23.505173 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr98t\" (UniqueName: \"kubernetes.io/projected/60a33d48-25bd-4b1c-91da-911e825718c6-kube-api-access-fr98t\") pod \"maas-api-revoke-sub-be167f9a-cldkf\" (UID: \"60a33d48-25bd-4b1c-91da-911e825718c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be167f9a-cldkf" Nov 03 20:56:23.519689 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:23.518721 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9311f42a-6pdhs" podStartSLOduration=2.518702459 podStartE2EDuration="2.518702459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:56:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:56:23.510741432 +0000 UTC m=+1018.736061285" watchObservedRunningTime="2026-11-03 20:56:23.518702459 +0000 UTC m=+1018.744022252" Nov 03 20:56:23.712847 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:23.712751 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b319d281-57hgb" Nov 03 20:56:23.740285 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:23.740251 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be167f9a-cldkf" Nov 03 20:56:23.815969 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:23.815177 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65e231f1-20dc-4c54-b7f1-4217ab1a0965-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65e231f1-20dc-4c54-b7f1-4217ab1a0965-maas-api-openshift-ca\") pod \"65e231f1-20dc-4c54-b7f1-4217ab1a0965\" (UID: \"65e231f1-20dc-4c54-b7f1-4217ab1a0965\") " Nov 03 20:56:23.815969 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:23.815221 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65e231f1-20dc-4c54-b7f1-4217ab1a0965-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65e231f1-20dc-4c54-b7f1-4217ab1a0965-maas-api-xks-ca\") pod \"65e231f1-20dc-4c54-b7f1-4217ab1a0965\" (UID: \"65e231f1-20dc-4c54-b7f1-4217ab1a0965\") " Nov 03 20:56:23.815969 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:23.815252 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65e231f1-20dc-4c54-b7f1-4217ab1a0965-kube-api-access-t8fjj\" (UniqueName: \"kubernetes.io/projected/65e231f1-20dc-4c54-b7f1-4217ab1a0965-kube-api-access-t8fjj\") pod \"65e231f1-20dc-4c54-b7f1-4217ab1a0965\" (UID: \"65e231f1-20dc-4c54-b7f1-4217ab1a0965\") " Nov 03 20:56:23.816586 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:23.816550 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65e231f1-20dc-4c54-b7f1-4217ab1a0965-maas-api-openshift-ca" pod "65e231f1-20dc-4c54-b7f1-4217ab1a0965" (UID: "65e231f1-20dc-4c54-b7f1-4217ab1a0965"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:56:23.817899 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:23.817878 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65e231f1-20dc-4c54-b7f1-4217ab1a0965-kube-api-access-t8fjj" pod "65e231f1-20dc-4c54-b7f1-4217ab1a0965" (UID: "65e231f1-20dc-4c54-b7f1-4217ab1a0965"). InnerVolumeSpecName "kube-api-access-t8fjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:56:23.818326 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:23.818304 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65e231f1-20dc-4c54-b7f1-4217ab1a0965-maas-api-xks-ca" pod "65e231f1-20dc-4c54-b7f1-4217ab1a0965" (UID: "65e231f1-20dc-4c54-b7f1-4217ab1a0965"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:56:23.913884 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:23.912872 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-588hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 20:56:23.913884 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:23.912928 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-588hz" podUID="327b41fd-c8a7-4926-bc59-8869ae87aa8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 20:56:23.913884 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:23.913861 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-59cjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 20:56:23.914095 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:23.913903 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-59cjd" podUID="a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 20:56:23.918238 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:23.918180 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65e231f1-20dc-4c54-b7f1-4217ab1a0965-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:56:23.918238 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:23.918210 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65e231f1-20dc-4c54-b7f1-4217ab1a0965-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:56:23.918518 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:23.918493 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8fjj\" (UniqueName: \"kubernetes.io/projected/65e231f1-20dc-4c54-b7f1-4217ab1a0965-kube-api-access-t8fjj\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:56:24.148367 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:24.148330 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71cb34d9-lmbgq"] Nov 03 20:56:24.148957 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:24.148942 4305 cpu_manager.go:414] "containerMap: removing container" podUID="65e231f1-20dc-4c54-b7f1-4217ab1a0965" containerName="revoke-keys" Nov 03 20:56:24.149021 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:24.148960 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65e231f1-20dc-4c54-b7f1-4217ab1a0965" containerName="revoke-keys" Nov 03 20:56:24.149077 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:24.149067 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="65e231f1-20dc-4c54-b7f1-4217ab1a0965" containerName="revoke-keys" Nov 03 20:56:24.166461 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:24.153864 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71cb34d9-lmbgq" Nov 03 20:56:24.206464 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:24.196485 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be167f9a-cldkf"] Nov 03 20:56:24.212791 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:24.212758 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71cb34d9-lmbgq"] Nov 03 20:56:24.324189 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:24.324153 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf7d3b6c-e2b5-4899-b04e-b78a14240c91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71cb34d9-lmbgq\" (UID: \"bf7d3b6c-e2b5-4899-b04e-b78a14240c91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71cb34d9-lmbgq" Nov 03 20:56:24.324373 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:24.324296 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf7d3b6c-e2b5-4899-b04e-b78a14240c91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71cb34d9-lmbgq\" (UID: \"bf7d3b6c-e2b5-4899-b04e-b78a14240c91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71cb34d9-lmbgq" Nov 03 20:56:24.324474 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:24.324361 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv2rd\" (UniqueName: \"kubernetes.io/projected/bf7d3b6c-e2b5-4899-b04e-b78a14240c91-kube-api-access-tv2rd\") pod \"maas-api-revoke-sub-71cb34d9-lmbgq\" (UID: \"bf7d3b6c-e2b5-4899-b04e-b78a14240c91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71cb34d9-lmbgq" Nov 03 20:56:24.382495 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:24.376455 4305 generic.go:356] "Generic (PLEG): container finished" podID="17a26a30-1c68-44c7-ac11-2269b4d2b3d4" containerID="d9814b638bf2433354ec201446b0e221cba4994f028d9c4a448b177d0622bb02" exitCode=0 Nov 03 20:56:24.382495 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:24.376638 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9311f42a-6pdhs" event={"ID":"17a26a30-1c68-44c7-ac11-2269b4d2b3d4","Type":"ContainerDied","Data":"d9814b638bf2433354ec201446b0e221cba4994f028d9c4a448b177d0622bb02"} Nov 03 20:56:24.392466 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:24.385578 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b319d281-57hgb" event={"ID":"65e231f1-20dc-4c54-b7f1-4217ab1a0965","Type":"ContainerDied","Data":"f531edcd873d032c46e9dba818a0be2bc31464866f2e2b8a8bf97887cedb0aec"} Nov 03 20:56:24.392466 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:24.385656 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f531edcd873d032c46e9dba818a0be2bc31464866f2e2b8a8bf97887cedb0aec" Nov 03 20:56:24.392466 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:24.385768 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b319d281-57hgb" Nov 03 20:56:24.414502 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:24.413513 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be167f9a-cldkf" event={"ID":"60a33d48-25bd-4b1c-91da-911e825718c6","Type":"ContainerStarted","Data":"45e051184b0b192c043dce2b90bbfd2bdac7ed60133579bc6e2ebf5707c8f213"} Nov 03 20:56:24.425442 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:24.425405 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf7d3b6c-e2b5-4899-b04e-b78a14240c91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71cb34d9-lmbgq\" (UID: \"bf7d3b6c-e2b5-4899-b04e-b78a14240c91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71cb34d9-lmbgq" Nov 03 20:56:24.425569 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:24.425513 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf7d3b6c-e2b5-4899-b04e-b78a14240c91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71cb34d9-lmbgq\" (UID: \"bf7d3b6c-e2b5-4899-b04e-b78a14240c91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71cb34d9-lmbgq" Nov 03 20:56:24.425569 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:24.425533 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv2rd\" (UniqueName: \"kubernetes.io/projected/bf7d3b6c-e2b5-4899-b04e-b78a14240c91-kube-api-access-tv2rd\") pod \"maas-api-revoke-sub-71cb34d9-lmbgq\" (UID: \"bf7d3b6c-e2b5-4899-b04e-b78a14240c91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71cb34d9-lmbgq" Nov 03 20:56:24.426107 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:24.426086 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf7d3b6c-e2b5-4899-b04e-b78a14240c91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71cb34d9-lmbgq\" (UID: \"bf7d3b6c-e2b5-4899-b04e-b78a14240c91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71cb34d9-lmbgq" Nov 03 20:56:24.428020 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:24.427996 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf7d3b6c-e2b5-4899-b04e-b78a14240c91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71cb34d9-lmbgq\" (UID: \"bf7d3b6c-e2b5-4899-b04e-b78a14240c91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71cb34d9-lmbgq" Nov 03 20:56:24.507577 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:24.507509 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv2rd\" (UniqueName: \"kubernetes.io/projected/bf7d3b6c-e2b5-4899-b04e-b78a14240c91-kube-api-access-tv2rd\") pod \"maas-api-revoke-sub-71cb34d9-lmbgq\" (UID: \"bf7d3b6c-e2b5-4899-b04e-b78a14240c91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71cb34d9-lmbgq" Nov 03 20:56:24.515994 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:24.515737 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71cb34d9-lmbgq" Nov 03 20:56:24.876459 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:24.875923 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-588hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 20:56:24.876459 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:24.875992 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-588hz" podUID="327b41fd-c8a7-4926-bc59-8869ae87aa8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 20:56:24.905458 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:24.904872 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-59cjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 20:56:24.905458 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:24.904914 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-59cjd" podUID="a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 20:56:24.995331 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:24.995229 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71cb34d9-lmbgq"] Nov 03 20:56:25.006919 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:56:25.006710 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf7d3b6c_e2b5_4899_b04e_b78a14240c91.slice/crio-da45d07045890c777857d09b6619d92567c1c2be054590ad9eb29ff55ad3aaa8 WatchSource:0}: Error finding container da45d07045890c777857d09b6619d92567c1c2be054590ad9eb29ff55ad3aaa8: Status 404 returned error can't find the container with id da45d07045890c777857d09b6619d92567c1c2be054590ad9eb29ff55ad3aaa8 Nov 03 20:56:25.422011 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:25.421973 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be167f9a-cldkf" event={"ID":"60a33d48-25bd-4b1c-91da-911e825718c6","Type":"ContainerStarted","Data":"95430a1363f89f9b8392141dba66bfcc0e0db818b4ec518d8e812c596e3d7df1"} Nov 03 20:56:25.423244 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:25.423224 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71cb34d9-lmbgq" event={"ID":"bf7d3b6c-e2b5-4899-b04e-b78a14240c91","Type":"ContainerStarted","Data":"da45d07045890c777857d09b6619d92567c1c2be054590ad9eb29ff55ad3aaa8"} Nov 03 20:56:25.616662 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:25.616348 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be167f9a-cldkf" podStartSLOduration=2.616326598 podStartE2EDuration="2.616326598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:56:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:56:25.547873156 +0000 UTC m=+1020.773192974" watchObservedRunningTime="2026-11-03 20:56:25.616326598 +0000 UTC m=+1020.841646392" Nov 03 20:56:25.733353 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:25.733325 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9311f42a-6pdhs" Nov 03 20:56:25.842885 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:25.842845 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17a26a30-1c68-44c7-ac11-2269b4d2b3d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17a26a30-1c68-44c7-ac11-2269b4d2b3d4-maas-api-openshift-ca\") pod \"17a26a30-1c68-44c7-ac11-2269b4d2b3d4\" (UID: \"17a26a30-1c68-44c7-ac11-2269b4d2b3d4\") " Nov 03 20:56:25.842885 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:25.842885 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17a26a30-1c68-44c7-ac11-2269b4d2b3d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17a26a30-1c68-44c7-ac11-2269b4d2b3d4-maas-api-xks-ca\") pod \"17a26a30-1c68-44c7-ac11-2269b4d2b3d4\" (UID: \"17a26a30-1c68-44c7-ac11-2269b4d2b3d4\") " Nov 03 20:56:25.843150 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:25.843022 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17a26a30-1c68-44c7-ac11-2269b4d2b3d4-kube-api-access-dn8p4\" (UniqueName: \"kubernetes.io/projected/17a26a30-1c68-44c7-ac11-2269b4d2b3d4-kube-api-access-dn8p4\") pod \"17a26a30-1c68-44c7-ac11-2269b4d2b3d4\" (UID: \"17a26a30-1c68-44c7-ac11-2269b4d2b3d4\") " Nov 03 20:56:25.843396 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:25.843355 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17a26a30-1c68-44c7-ac11-2269b4d2b3d4-maas-api-openshift-ca" pod "17a26a30-1c68-44c7-ac11-2269b4d2b3d4" (UID: "17a26a30-1c68-44c7-ac11-2269b4d2b3d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:56:25.845412 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:25.845387 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17a26a30-1c68-44c7-ac11-2269b4d2b3d4-kube-api-access-dn8p4" pod "17a26a30-1c68-44c7-ac11-2269b4d2b3d4" (UID: "17a26a30-1c68-44c7-ac11-2269b4d2b3d4"). InnerVolumeSpecName "kube-api-access-dn8p4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:56:25.845502 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:25.845488 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17a26a30-1c68-44c7-ac11-2269b4d2b3d4-maas-api-xks-ca" pod "17a26a30-1c68-44c7-ac11-2269b4d2b3d4" (UID: "17a26a30-1c68-44c7-ac11-2269b4d2b3d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:56:25.910270 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:25.909579 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-59cjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 20:56:25.910270 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:25.909631 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-59cjd" podUID="a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 20:56:25.910270 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:25.909975 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-588hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 20:56:25.910270 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:25.909997 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-588hz" podUID="327b41fd-c8a7-4926-bc59-8869ae87aa8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 20:56:25.945219 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:25.945178 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dn8p4\" (UniqueName: \"kubernetes.io/projected/17a26a30-1c68-44c7-ac11-2269b4d2b3d4-kube-api-access-dn8p4\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:56:25.945424 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:25.945216 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17a26a30-1c68-44c7-ac11-2269b4d2b3d4-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:56:25.945424 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:25.945247 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17a26a30-1c68-44c7-ac11-2269b4d2b3d4-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:56:26.012559 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:26.012527 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj"] Nov 03 20:56:26.013187 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:26.013171 4305 cpu_manager.go:414] "containerMap: removing container" podUID="17a26a30-1c68-44c7-ac11-2269b4d2b3d4" containerName="revoke-keys" Nov 03 20:56:26.013229 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:26.013188 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17a26a30-1c68-44c7-ac11-2269b4d2b3d4" containerName="revoke-keys" Nov 03 20:56:26.013297 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:26.013289 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="17a26a30-1c68-44c7-ac11-2269b4d2b3d4" containerName="revoke-keys" Nov 03 20:56:26.016742 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:26.016719 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" Nov 03 20:56:26.034282 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:26.034250 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj"] Nov 03 20:56:26.045968 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:26.045936 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8793f095-97b1-4e0a-8382-b541f90a0de2-home\") pod \"distinct-w3-89bb58-kserve-5cb775b97b-pvwsj\" (UID: \"8793f095-97b1-4e0a-8382-b541f90a0de2\") " pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" Nov 03 20:56:26.045968 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:26.045972 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8793f095-97b1-4e0a-8382-b541f90a0de2-tls-certs\") pod \"distinct-w3-89bb58-kserve-5cb775b97b-pvwsj\" (UID: \"8793f095-97b1-4e0a-8382-b541f90a0de2\") " pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" Nov 03 20:56:26.046243 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:26.045997 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8793f095-97b1-4e0a-8382-b541f90a0de2-model-cache\") pod \"distinct-w3-89bb58-kserve-5cb775b97b-pvwsj\" (UID: \"8793f095-97b1-4e0a-8382-b541f90a0de2\") " pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" Nov 03 20:56:26.046243 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:26.046226 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47bjp\" (UniqueName: \"kubernetes.io/projected/8793f095-97b1-4e0a-8382-b541f90a0de2-kube-api-access-47bjp\") pod \"distinct-w3-89bb58-kserve-5cb775b97b-pvwsj\" (UID: \"8793f095-97b1-4e0a-8382-b541f90a0de2\") " pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" Nov 03 20:56:26.046369 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:26.046269 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8793f095-97b1-4e0a-8382-b541f90a0de2-tmp-dir\") pod \"distinct-w3-89bb58-kserve-5cb775b97b-pvwsj\" (UID: \"8793f095-97b1-4e0a-8382-b541f90a0de2\") " pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" Nov 03 20:56:26.046369 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:26.046359 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8793f095-97b1-4e0a-8382-b541f90a0de2-dshm\") pod \"distinct-w3-89bb58-kserve-5cb775b97b-pvwsj\" (UID: \"8793f095-97b1-4e0a-8382-b541f90a0de2\") " pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" Nov 03 20:56:26.099574 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:26.099543 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-89bb58\"/\"distinct-w3-89bb58-kserve-self-signed-certs\"" Nov 03 20:56:26.147080 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:26.147035 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47bjp\" (UniqueName: \"kubernetes.io/projected/8793f095-97b1-4e0a-8382-b541f90a0de2-kube-api-access-47bjp\") pod \"distinct-w3-89bb58-kserve-5cb775b97b-pvwsj\" (UID: \"8793f095-97b1-4e0a-8382-b541f90a0de2\") " pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" Nov 03 20:56:26.147080 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:26.147083 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8793f095-97b1-4e0a-8382-b541f90a0de2-tmp-dir\") pod \"distinct-w3-89bb58-kserve-5cb775b97b-pvwsj\" (UID: \"8793f095-97b1-4e0a-8382-b541f90a0de2\") " pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" Nov 03 20:56:26.147306 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:26.147143 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8793f095-97b1-4e0a-8382-b541f90a0de2-dshm\") pod \"distinct-w3-89bb58-kserve-5cb775b97b-pvwsj\" (UID: \"8793f095-97b1-4e0a-8382-b541f90a0de2\") " pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" Nov 03 20:56:26.147306 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:26.147177 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8793f095-97b1-4e0a-8382-b541f90a0de2-home\") pod \"distinct-w3-89bb58-kserve-5cb775b97b-pvwsj\" (UID: \"8793f095-97b1-4e0a-8382-b541f90a0de2\") " pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" Nov 03 20:56:26.147306 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:26.147195 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8793f095-97b1-4e0a-8382-b541f90a0de2-tls-certs\") pod \"distinct-w3-89bb58-kserve-5cb775b97b-pvwsj\" (UID: \"8793f095-97b1-4e0a-8382-b541f90a0de2\") " pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" Nov 03 20:56:26.147306 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:26.147217 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8793f095-97b1-4e0a-8382-b541f90a0de2-model-cache\") pod \"distinct-w3-89bb58-kserve-5cb775b97b-pvwsj\" (UID: \"8793f095-97b1-4e0a-8382-b541f90a0de2\") " pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" Nov 03 20:56:26.147772 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:26.147691 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8793f095-97b1-4e0a-8382-b541f90a0de2-model-cache\") pod \"distinct-w3-89bb58-kserve-5cb775b97b-pvwsj\" (UID: \"8793f095-97b1-4e0a-8382-b541f90a0de2\") " pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" Nov 03 20:56:26.147772 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:26.147739 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8793f095-97b1-4e0a-8382-b541f90a0de2-home\") pod \"distinct-w3-89bb58-kserve-5cb775b97b-pvwsj\" (UID: \"8793f095-97b1-4e0a-8382-b541f90a0de2\") " pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" Nov 03 20:56:26.147953 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:26.147926 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8793f095-97b1-4e0a-8382-b541f90a0de2-tmp-dir\") pod \"distinct-w3-89bb58-kserve-5cb775b97b-pvwsj\" (UID: \"8793f095-97b1-4e0a-8382-b541f90a0de2\") " pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" Nov 03 20:56:26.150095 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:26.150077 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8793f095-97b1-4e0a-8382-b541f90a0de2-dshm\") pod \"distinct-w3-89bb58-kserve-5cb775b97b-pvwsj\" (UID: \"8793f095-97b1-4e0a-8382-b541f90a0de2\") " pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" Nov 03 20:56:26.150224 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:26.150111 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8793f095-97b1-4e0a-8382-b541f90a0de2-tls-certs\") pod \"distinct-w3-89bb58-kserve-5cb775b97b-pvwsj\" (UID: \"8793f095-97b1-4e0a-8382-b541f90a0de2\") " pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" Nov 03 20:56:26.218558 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:26.214256 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47bjp\" (UniqueName: \"kubernetes.io/projected/8793f095-97b1-4e0a-8382-b541f90a0de2-kube-api-access-47bjp\") pod \"distinct-w3-89bb58-kserve-5cb775b97b-pvwsj\" (UID: \"8793f095-97b1-4e0a-8382-b541f90a0de2\") " pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" Nov 03 20:56:26.333850 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:26.333753 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" Nov 03 20:56:26.450799 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:26.450765 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9311f42a-6pdhs" Nov 03 20:56:26.451194 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:26.450814 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9311f42a-6pdhs" event={"ID":"17a26a30-1c68-44c7-ac11-2269b4d2b3d4","Type":"ContainerDied","Data":"36ccc5d4882d9339e78509c6a509ed7a85c207549ea7ef882c4bf956a87c8b71"} Nov 03 20:56:26.451194 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:26.450867 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36ccc5d4882d9339e78509c6a509ed7a85c207549ea7ef882c4bf956a87c8b71" Nov 03 20:56:26.502543 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:26.480487 4305 generic.go:356] "Generic (PLEG): container finished" podID="60a33d48-25bd-4b1c-91da-911e825718c6" containerID="95430a1363f89f9b8392141dba66bfcc0e0db818b4ec518d8e812c596e3d7df1" exitCode=0 Nov 03 20:56:26.502543 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:26.480607 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be167f9a-cldkf" event={"ID":"60a33d48-25bd-4b1c-91da-911e825718c6","Type":"ContainerDied","Data":"95430a1363f89f9b8392141dba66bfcc0e0db818b4ec518d8e812c596e3d7df1"} Nov 03 20:56:26.513279 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:26.513155 4305 generic.go:356] "Generic (PLEG): container finished" podID="bf7d3b6c-e2b5-4899-b04e-b78a14240c91" containerID="285fcb1b85714e7ef017771157e327968cff1d0b9cfbfff97bb4b38e92ddb58a" exitCode=0 Nov 03 20:56:26.513279 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:26.513213 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71cb34d9-lmbgq" event={"ID":"bf7d3b6c-e2b5-4899-b04e-b78a14240c91","Type":"ContainerDied","Data":"285fcb1b85714e7ef017771157e327968cff1d0b9cfbfff97bb4b38e92ddb58a"} Nov 03 20:56:26.847225 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:26.847194 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj"] Nov 03 20:56:26.848043 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:56:26.848011 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8793f095_97b1_4e0a_8382_b541f90a0de2.slice/crio-d8f79b1fd43b37216ba0c6131eeea5e5480b2b73e8db18e2be7500e0c94ce5d0 WatchSource:0}: Error finding container d8f79b1fd43b37216ba0c6131eeea5e5480b2b73e8db18e2be7500e0c94ce5d0: Status 404 returned error can't find the container with id d8f79b1fd43b37216ba0c6131eeea5e5480b2b73e8db18e2be7500e0c94ce5d0 Nov 03 20:56:26.870309 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:26.861848 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-588hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 20:56:26.870309 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:26.861907 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-588hz" podUID="327b41fd-c8a7-4926-bc59-8869ae87aa8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 20:56:26.876979 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:26.876943 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-59cjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 20:56:26.877145 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:26.877037 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-59cjd" podUID="a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 20:56:27.523450 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:27.523402 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" event={"ID":"8793f095-97b1-4e0a-8382-b541f90a0de2","Type":"ContainerStarted","Data":"757468dfac9960a8b7021a8d33be8a19cc19b93f4d0952d92a933ed78d858d01"} Nov 03 20:56:27.523975 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:27.523470 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" event={"ID":"8793f095-97b1-4e0a-8382-b541f90a0de2","Type":"ContainerStarted","Data":"d8f79b1fd43b37216ba0c6131eeea5e5480b2b73e8db18e2be7500e0c94ce5d0"} Nov 03 20:56:27.649683 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:27.647558 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" podStartSLOduration=2.647538334 podStartE2EDuration="2.647538334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:56:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:56:27.643740709 +0000 UTC m=+1022.869060499" watchObservedRunningTime="2026-11-03 20:56:27.647538334 +0000 UTC m=+1022.872858125" Nov 03 20:56:27.709867 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:27.709836 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71cb34d9-lmbgq" Nov 03 20:56:27.750149 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:27.750123 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be167f9a-cldkf" Nov 03 20:56:27.799782 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:27.799701 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60a33d48-25bd-4b1c-91da-911e825718c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60a33d48-25bd-4b1c-91da-911e825718c6-maas-api-xks-ca\") pod \"60a33d48-25bd-4b1c-91da-911e825718c6\" (UID: \"60a33d48-25bd-4b1c-91da-911e825718c6\") " Nov 03 20:56:27.799782 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:27.799757 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf7d3b6c-e2b5-4899-b04e-b78a14240c91-kube-api-access-tv2rd\" (UniqueName: \"kubernetes.io/projected/bf7d3b6c-e2b5-4899-b04e-b78a14240c91-kube-api-access-tv2rd\") pod \"bf7d3b6c-e2b5-4899-b04e-b78a14240c91\" (UID: \"bf7d3b6c-e2b5-4899-b04e-b78a14240c91\") " Nov 03 20:56:27.800063 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:27.799794 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60a33d48-25bd-4b1c-91da-911e825718c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60a33d48-25bd-4b1c-91da-911e825718c6-maas-api-openshift-ca\") pod \"60a33d48-25bd-4b1c-91da-911e825718c6\" (UID: \"60a33d48-25bd-4b1c-91da-911e825718c6\") " Nov 03 20:56:27.800063 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:27.799895 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf7d3b6c-e2b5-4899-b04e-b78a14240c91-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf7d3b6c-e2b5-4899-b04e-b78a14240c91-maas-api-xks-ca\") pod \"bf7d3b6c-e2b5-4899-b04e-b78a14240c91\" (UID: \"bf7d3b6c-e2b5-4899-b04e-b78a14240c91\") " Nov 03 20:56:27.800063 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:27.800014 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60a33d48-25bd-4b1c-91da-911e825718c6-kube-api-access-fr98t\" (UniqueName: \"kubernetes.io/projected/60a33d48-25bd-4b1c-91da-911e825718c6-kube-api-access-fr98t\") pod \"60a33d48-25bd-4b1c-91da-911e825718c6\" (UID: \"60a33d48-25bd-4b1c-91da-911e825718c6\") " Nov 03 20:56:27.800234 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:27.800102 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf7d3b6c-e2b5-4899-b04e-b78a14240c91-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf7d3b6c-e2b5-4899-b04e-b78a14240c91-maas-api-openshift-ca\") pod \"bf7d3b6c-e2b5-4899-b04e-b78a14240c91\" (UID: \"bf7d3b6c-e2b5-4899-b04e-b78a14240c91\") " Nov 03 20:56:27.800894 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:27.800576 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60a33d48-25bd-4b1c-91da-911e825718c6-maas-api-openshift-ca" pod "60a33d48-25bd-4b1c-91da-911e825718c6" (UID: "60a33d48-25bd-4b1c-91da-911e825718c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:56:27.801024 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:27.800897 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf7d3b6c-e2b5-4899-b04e-b78a14240c91-maas-api-openshift-ca" pod "bf7d3b6c-e2b5-4899-b04e-b78a14240c91" (UID: "bf7d3b6c-e2b5-4899-b04e-b78a14240c91"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:56:27.801253 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:27.801220 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60a33d48-25bd-4b1c-91da-911e825718c6-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:56:27.802541 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:27.802516 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf7d3b6c-e2b5-4899-b04e-b78a14240c91-maas-api-xks-ca" pod "bf7d3b6c-e2b5-4899-b04e-b78a14240c91" (UID: "bf7d3b6c-e2b5-4899-b04e-b78a14240c91"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:56:27.802690 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:27.802674 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60a33d48-25bd-4b1c-91da-911e825718c6-maas-api-xks-ca" pod "60a33d48-25bd-4b1c-91da-911e825718c6" (UID: "60a33d48-25bd-4b1c-91da-911e825718c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:56:27.803650 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:27.803633 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf7d3b6c-e2b5-4899-b04e-b78a14240c91-kube-api-access-tv2rd" pod "bf7d3b6c-e2b5-4899-b04e-b78a14240c91" (UID: "bf7d3b6c-e2b5-4899-b04e-b78a14240c91"). InnerVolumeSpecName "kube-api-access-tv2rd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:56:27.803715 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:27.803696 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60a33d48-25bd-4b1c-91da-911e825718c6-kube-api-access-fr98t" pod "60a33d48-25bd-4b1c-91da-911e825718c6" (UID: "60a33d48-25bd-4b1c-91da-911e825718c6"). InnerVolumeSpecName "kube-api-access-fr98t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:56:27.858795 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:27.858755 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-588hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 20:56:27.859003 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:27.858813 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-588hz" podUID="327b41fd-c8a7-4926-bc59-8869ae87aa8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 20:56:27.877522 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:27.877476 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-59cjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 03 20:56:27.877717 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:27.877551 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-59cjd" podUID="a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 03 20:56:27.902449 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:27.902398 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf7d3b6c-e2b5-4899-b04e-b78a14240c91-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:56:27.902449 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:27.902425 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr98t\" (UniqueName: \"kubernetes.io/projected/60a33d48-25bd-4b1c-91da-911e825718c6-kube-api-access-fr98t\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:56:27.902660 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:27.902465 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf7d3b6c-e2b5-4899-b04e-b78a14240c91-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:56:27.902660 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:27.902475 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60a33d48-25bd-4b1c-91da-911e825718c6-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:56:27.902660 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:27.902484 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tv2rd\" (UniqueName: \"kubernetes.io/projected/bf7d3b6c-e2b5-4899-b04e-b78a14240c91-kube-api-access-tv2rd\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:56:28.334742 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:28.334710 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" Nov 03 20:56:28.545065 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:28.545030 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be167f9a-cldkf" event={"ID":"60a33d48-25bd-4b1c-91da-911e825718c6","Type":"ContainerDied","Data":"45e051184b0b192c043dce2b90bbfd2bdac7ed60133579bc6e2ebf5707c8f213"} Nov 03 20:56:28.545642 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:28.545624 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45e051184b0b192c043dce2b90bbfd2bdac7ed60133579bc6e2ebf5707c8f213" Nov 03 20:56:28.545763 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:28.545044 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be167f9a-cldkf" Nov 03 20:56:28.547301 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:28.547267 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71cb34d9-lmbgq" event={"ID":"bf7d3b6c-e2b5-4899-b04e-b78a14240c91","Type":"ContainerDied","Data":"da45d07045890c777857d09b6619d92567c1c2be054590ad9eb29ff55ad3aaa8"} Nov 03 20:56:28.547400 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:28.547304 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71cb34d9-lmbgq" Nov 03 20:56:28.547400 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:28.547326 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da45d07045890c777857d09b6619d92567c1c2be054590ad9eb29ff55ad3aaa8" Nov 03 20:56:28.858789 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:28.858749 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-588hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 03 20:56:28.858977 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:28.858814 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-588hz" podUID="327b41fd-c8a7-4926-bc59-8869ae87aa8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 03 20:56:29.199685 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:29.199650 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-59cjd" Nov 03 20:56:29.200167 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:29.200142 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-59cjd" Nov 03 20:56:29.201223 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:29.201200 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-59cjd" Nov 03 20:56:30.022559 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:30.022523 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-588hz" Nov 03 20:56:30.023165 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:30.023150 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-588hz" Nov 03 20:56:30.024018 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:30.023998 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-588hz" Nov 03 20:56:30.065662 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:30.065632 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2b3bf0e-6xhtq"] Nov 03 20:56:30.066204 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:30.066187 4305 cpu_manager.go:414] "containerMap: removing container" podUID="bf7d3b6c-e2b5-4899-b04e-b78a14240c91" containerName="revoke-keys" Nov 03 20:56:30.066204 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:30.066203 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf7d3b6c-e2b5-4899-b04e-b78a14240c91" containerName="revoke-keys" Nov 03 20:56:30.066383 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:30.066230 4305 cpu_manager.go:414] "containerMap: removing container" podUID="60a33d48-25bd-4b1c-91da-911e825718c6" containerName="revoke-keys" Nov 03 20:56:30.066383 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:30.066235 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60a33d48-25bd-4b1c-91da-911e825718c6" containerName="revoke-keys" Nov 03 20:56:30.066383 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:30.066312 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="60a33d48-25bd-4b1c-91da-911e825718c6" containerName="revoke-keys" Nov 03 20:56:30.066383 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:30.066324 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf7d3b6c-e2b5-4899-b04e-b78a14240c91" containerName="revoke-keys" Nov 03 20:56:30.070114 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:30.070087 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b3bf0e-6xhtq" Nov 03 20:56:30.127069 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:30.127038 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2b3bf0e-6xhtq"] Nov 03 20:56:30.127314 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:30.127300 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:56:30.127376 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:30.127363 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:56:30.127427 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:30.127411 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:56:30.127492 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:30.127426 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77vtd\"" Nov 03 20:56:30.133012 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:30.132985 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hwbf\" (UniqueName: \"kubernetes.io/projected/6cdbecf8-2c06-48d9-8f80-9ca469289354-kube-api-access-2hwbf\") pod \"maas-api-revoke-sub-c2b3bf0e-6xhtq\" (UID: \"6cdbecf8-2c06-48d9-8f80-9ca469289354\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b3bf0e-6xhtq" Nov 03 20:56:30.133184 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:30.133141 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cdbecf8-2c06-48d9-8f80-9ca469289354-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2b3bf0e-6xhtq\" (UID: \"6cdbecf8-2c06-48d9-8f80-9ca469289354\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b3bf0e-6xhtq" Nov 03 20:56:30.133292 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:30.133267 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cdbecf8-2c06-48d9-8f80-9ca469289354-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2b3bf0e-6xhtq\" (UID: \"6cdbecf8-2c06-48d9-8f80-9ca469289354\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b3bf0e-6xhtq" Nov 03 20:56:30.235032 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:30.235001 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cdbecf8-2c06-48d9-8f80-9ca469289354-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2b3bf0e-6xhtq\" (UID: \"6cdbecf8-2c06-48d9-8f80-9ca469289354\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b3bf0e-6xhtq" Nov 03 20:56:30.235219 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:30.235084 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cdbecf8-2c06-48d9-8f80-9ca469289354-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2b3bf0e-6xhtq\" (UID: \"6cdbecf8-2c06-48d9-8f80-9ca469289354\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b3bf0e-6xhtq" Nov 03 20:56:30.235219 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:30.235176 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hwbf\" (UniqueName: \"kubernetes.io/projected/6cdbecf8-2c06-48d9-8f80-9ca469289354-kube-api-access-2hwbf\") pod \"maas-api-revoke-sub-c2b3bf0e-6xhtq\" (UID: \"6cdbecf8-2c06-48d9-8f80-9ca469289354\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b3bf0e-6xhtq" Nov 03 20:56:30.235683 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:30.235663 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cdbecf8-2c06-48d9-8f80-9ca469289354-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2b3bf0e-6xhtq\" (UID: \"6cdbecf8-2c06-48d9-8f80-9ca469289354\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b3bf0e-6xhtq" Nov 03 20:56:30.237635 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:30.237613 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cdbecf8-2c06-48d9-8f80-9ca469289354-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2b3bf0e-6xhtq\" (UID: \"6cdbecf8-2c06-48d9-8f80-9ca469289354\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b3bf0e-6xhtq" Nov 03 20:56:30.297136 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:30.296678 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hwbf\" (UniqueName: \"kubernetes.io/projected/6cdbecf8-2c06-48d9-8f80-9ca469289354-kube-api-access-2hwbf\") pod \"maas-api-revoke-sub-c2b3bf0e-6xhtq\" (UID: \"6cdbecf8-2c06-48d9-8f80-9ca469289354\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b3bf0e-6xhtq" Nov 03 20:56:30.385117 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:30.385088 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b3bf0e-6xhtq" Nov 03 20:56:30.462777 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:30.462749 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a02f5a1-c72m7"] Nov 03 20:56:30.469962 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:30.469937 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a02f5a1-c72m7" Nov 03 20:56:30.522766 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:30.522716 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a02f5a1-c72m7"] Nov 03 20:56:30.540229 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:30.540187 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhh76\" (UniqueName: \"kubernetes.io/projected/97f3c196-0376-41ae-8b28-930d728277dd-kube-api-access-lhh76\") pod \"maas-api-revoke-sub-7a02f5a1-c72m7\" (UID: \"97f3c196-0376-41ae-8b28-930d728277dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a02f5a1-c72m7" Nov 03 20:56:30.540424 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:30.540365 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97f3c196-0376-41ae-8b28-930d728277dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a02f5a1-c72m7\" (UID: \"97f3c196-0376-41ae-8b28-930d728277dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a02f5a1-c72m7" Nov 03 20:56:30.540424 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:30.540410 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97f3c196-0376-41ae-8b28-930d728277dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a02f5a1-c72m7\" (UID: \"97f3c196-0376-41ae-8b28-930d728277dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a02f5a1-c72m7" Nov 03 20:56:30.641964 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:30.641869 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97f3c196-0376-41ae-8b28-930d728277dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a02f5a1-c72m7\" (UID: \"97f3c196-0376-41ae-8b28-930d728277dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a02f5a1-c72m7" Nov 03 20:56:30.641964 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:30.641933 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97f3c196-0376-41ae-8b28-930d728277dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a02f5a1-c72m7\" (UID: \"97f3c196-0376-41ae-8b28-930d728277dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a02f5a1-c72m7" Nov 03 20:56:30.642474 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:30.642426 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhh76\" (UniqueName: \"kubernetes.io/projected/97f3c196-0376-41ae-8b28-930d728277dd-kube-api-access-lhh76\") pod \"maas-api-revoke-sub-7a02f5a1-c72m7\" (UID: \"97f3c196-0376-41ae-8b28-930d728277dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a02f5a1-c72m7" Nov 03 20:56:30.642764 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:30.642743 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97f3c196-0376-41ae-8b28-930d728277dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a02f5a1-c72m7\" (UID: \"97f3c196-0376-41ae-8b28-930d728277dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a02f5a1-c72m7" Nov 03 20:56:30.644648 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:30.644622 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97f3c196-0376-41ae-8b28-930d728277dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a02f5a1-c72m7\" (UID: \"97f3c196-0376-41ae-8b28-930d728277dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a02f5a1-c72m7" Nov 03 20:56:30.703722 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:30.703689 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhh76\" (UniqueName: \"kubernetes.io/projected/97f3c196-0376-41ae-8b28-930d728277dd-kube-api-access-lhh76\") pod \"maas-api-revoke-sub-7a02f5a1-c72m7\" (UID: \"97f3c196-0376-41ae-8b28-930d728277dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a02f5a1-c72m7" Nov 03 20:56:30.769917 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:30.769888 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2b3bf0e-6xhtq"] Nov 03 20:56:30.770674 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:56:30.770643 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cdbecf8_2c06_48d9_8f80_9ca469289354.slice/crio-b4c29fbfdc6cdebcabdb5f50ae20cb30cbe44d338a1944cca3505047f192e0a4 WatchSource:0}: Error finding container b4c29fbfdc6cdebcabdb5f50ae20cb30cbe44d338a1944cca3505047f192e0a4: Status 404 returned error can't find the container with id b4c29fbfdc6cdebcabdb5f50ae20cb30cbe44d338a1944cca3505047f192e0a4 Nov 03 20:56:30.786771 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:30.786740 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a02f5a1-c72m7" Nov 03 20:56:31.164346 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:31.164318 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a02f5a1-c72m7"] Nov 03 20:56:31.164706 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:56:31.164635 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97f3c196_0376_41ae_8b28_930d728277dd.slice/crio-bbb2a21f0936e321755063ab9e63a690b743f24a8f4b82e026c9db3a692ba243 WatchSource:0}: Error finding container bbb2a21f0936e321755063ab9e63a690b743f24a8f4b82e026c9db3a692ba243: Status 404 returned error can't find the container with id bbb2a21f0936e321755063ab9e63a690b743f24a8f4b82e026c9db3a692ba243 Nov 03 20:56:31.572341 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:31.572299 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b3bf0e-6xhtq" event={"ID":"6cdbecf8-2c06-48d9-8f80-9ca469289354","Type":"ContainerStarted","Data":"a3337c5cae7ab4b379111ff3148e80cb85402233c117e1eff1e03cd1b2070424"} Nov 03 20:56:31.572570 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:31.572395 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b3bf0e-6xhtq" event={"ID":"6cdbecf8-2c06-48d9-8f80-9ca469289354","Type":"ContainerStarted","Data":"b4c29fbfdc6cdebcabdb5f50ae20cb30cbe44d338a1944cca3505047f192e0a4"} Nov 03 20:56:31.573955 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:31.573927 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a02f5a1-c72m7" event={"ID":"97f3c196-0376-41ae-8b28-930d728277dd","Type":"ContainerStarted","Data":"453723776d937db36be3b89f44a51b136ec711c3ac982f8ef384e203e93358bf"} Nov 03 20:56:31.574056 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:31.573976 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a02f5a1-c72m7" event={"ID":"97f3c196-0376-41ae-8b28-930d728277dd","Type":"ContainerStarted","Data":"bbb2a21f0936e321755063ab9e63a690b743f24a8f4b82e026c9db3a692ba243"} Nov 03 20:56:31.689061 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:31.689011 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b3bf0e-6xhtq" podStartSLOduration=2.688992689 podStartE2EDuration="2.688992689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:56:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:56:31.686901196 +0000 UTC m=+1026.912221012" watchObservedRunningTime="2026-11-03 20:56:31.688992689 +0000 UTC m=+1026.914312479" Nov 03 20:56:31.804559 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:31.804499 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a02f5a1-c72m7" podStartSLOduration=1.804479948 podStartE2EDuration="1.804479948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:56:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:56:31.801838557 +0000 UTC m=+1027.027158355" watchObservedRunningTime="2026-11-03 20:56:31.804479948 +0000 UTC m=+1027.029799740" Nov 03 20:56:32.582608 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:32.582525 4305 generic.go:356] "Generic (PLEG): container finished" podID="97f3c196-0376-41ae-8b28-930d728277dd" containerID="453723776d937db36be3b89f44a51b136ec711c3ac982f8ef384e203e93358bf" exitCode=0 Nov 03 20:56:32.583012 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:32.582607 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a02f5a1-c72m7" event={"ID":"97f3c196-0376-41ae-8b28-930d728277dd","Type":"ContainerDied","Data":"453723776d937db36be3b89f44a51b136ec711c3ac982f8ef384e203e93358bf"} Nov 03 20:56:32.584494 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:32.584471 4305 generic.go:356] "Generic (PLEG): container finished" podID="6cdbecf8-2c06-48d9-8f80-9ca469289354" containerID="a3337c5cae7ab4b379111ff3148e80cb85402233c117e1eff1e03cd1b2070424" exitCode=0 Nov 03 20:56:32.584589 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:32.584471 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b3bf0e-6xhtq" event={"ID":"6cdbecf8-2c06-48d9-8f80-9ca469289354","Type":"ContainerDied","Data":"a3337c5cae7ab4b379111ff3148e80cb85402233c117e1eff1e03cd1b2070424"} Nov 03 20:56:33.757817 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:33.757787 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b3bf0e-6xhtq" Nov 03 20:56:33.795778 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:33.795754 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a02f5a1-c72m7" Nov 03 20:56:33.879464 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:33.879375 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97f3c196-0376-41ae-8b28-930d728277dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97f3c196-0376-41ae-8b28-930d728277dd-maas-api-xks-ca\") pod \"97f3c196-0376-41ae-8b28-930d728277dd\" (UID: \"97f3c196-0376-41ae-8b28-930d728277dd\") " Nov 03 20:56:33.879629 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:33.879506 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cdbecf8-2c06-48d9-8f80-9ca469289354-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cdbecf8-2c06-48d9-8f80-9ca469289354-maas-api-xks-ca\") pod \"6cdbecf8-2c06-48d9-8f80-9ca469289354\" (UID: \"6cdbecf8-2c06-48d9-8f80-9ca469289354\") " Nov 03 20:56:33.879629 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:33.879541 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97f3c196-0376-41ae-8b28-930d728277dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97f3c196-0376-41ae-8b28-930d728277dd-maas-api-openshift-ca\") pod \"97f3c196-0376-41ae-8b28-930d728277dd\" (UID: \"97f3c196-0376-41ae-8b28-930d728277dd\") " Nov 03 20:56:33.879752 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:33.879627 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97f3c196-0376-41ae-8b28-930d728277dd-kube-api-access-lhh76\" (UniqueName: \"kubernetes.io/projected/97f3c196-0376-41ae-8b28-930d728277dd-kube-api-access-lhh76\") pod \"97f3c196-0376-41ae-8b28-930d728277dd\" (UID: \"97f3c196-0376-41ae-8b28-930d728277dd\") " Nov 03 20:56:33.879752 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:33.879679 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cdbecf8-2c06-48d9-8f80-9ca469289354-kube-api-access-2hwbf\" (UniqueName: \"kubernetes.io/projected/6cdbecf8-2c06-48d9-8f80-9ca469289354-kube-api-access-2hwbf\") pod \"6cdbecf8-2c06-48d9-8f80-9ca469289354\" (UID: \"6cdbecf8-2c06-48d9-8f80-9ca469289354\") " Nov 03 20:56:33.879878 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:33.879856 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cdbecf8-2c06-48d9-8f80-9ca469289354-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cdbecf8-2c06-48d9-8f80-9ca469289354-maas-api-openshift-ca\") pod \"6cdbecf8-2c06-48d9-8f80-9ca469289354\" (UID: \"6cdbecf8-2c06-48d9-8f80-9ca469289354\") " Nov 03 20:56:33.879954 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:33.879936 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97f3c196-0376-41ae-8b28-930d728277dd-maas-api-openshift-ca" pod "97f3c196-0376-41ae-8b28-930d728277dd" (UID: "97f3c196-0376-41ae-8b28-930d728277dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:56:33.880240 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:33.880204 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cdbecf8-2c06-48d9-8f80-9ca469289354-maas-api-openshift-ca" pod "6cdbecf8-2c06-48d9-8f80-9ca469289354" (UID: "6cdbecf8-2c06-48d9-8f80-9ca469289354"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:56:33.880997 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:33.880978 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97f3c196-0376-41ae-8b28-930d728277dd-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:56:33.881113 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:33.881098 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cdbecf8-2c06-48d9-8f80-9ca469289354-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:56:33.882355 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:33.882189 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cdbecf8-2c06-48d9-8f80-9ca469289354-maas-api-xks-ca" pod "6cdbecf8-2c06-48d9-8f80-9ca469289354" (UID: "6cdbecf8-2c06-48d9-8f80-9ca469289354"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:56:33.882546 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:33.882394 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97f3c196-0376-41ae-8b28-930d728277dd-maas-api-xks-ca" pod "97f3c196-0376-41ae-8b28-930d728277dd" (UID: "97f3c196-0376-41ae-8b28-930d728277dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:56:33.883597 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:33.883575 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97f3c196-0376-41ae-8b28-930d728277dd-kube-api-access-lhh76" pod "97f3c196-0376-41ae-8b28-930d728277dd" (UID: "97f3c196-0376-41ae-8b28-930d728277dd"). InnerVolumeSpecName "kube-api-access-lhh76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:56:33.883683 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:33.883608 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cdbecf8-2c06-48d9-8f80-9ca469289354-kube-api-access-2hwbf" pod "6cdbecf8-2c06-48d9-8f80-9ca469289354" (UID: "6cdbecf8-2c06-48d9-8f80-9ca469289354"). InnerVolumeSpecName "kube-api-access-2hwbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:56:33.982875 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:33.982836 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhh76\" (UniqueName: \"kubernetes.io/projected/97f3c196-0376-41ae-8b28-930d728277dd-kube-api-access-lhh76\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:56:33.982875 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:33.982865 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hwbf\" (UniqueName: \"kubernetes.io/projected/6cdbecf8-2c06-48d9-8f80-9ca469289354-kube-api-access-2hwbf\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:56:33.982875 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:33.982877 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97f3c196-0376-41ae-8b28-930d728277dd-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:56:33.982875 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:33.982887 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cdbecf8-2c06-48d9-8f80-9ca469289354-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:56:34.604805 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:34.604764 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a02f5a1-c72m7" event={"ID":"97f3c196-0376-41ae-8b28-930d728277dd","Type":"ContainerDied","Data":"bbb2a21f0936e321755063ab9e63a690b743f24a8f4b82e026c9db3a692ba243"} Nov 03 20:56:34.605025 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:34.604768 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a02f5a1-c72m7" Nov 03 20:56:34.605025 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:34.604857 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bbb2a21f0936e321755063ab9e63a690b743f24a8f4b82e026c9db3a692ba243" Nov 03 20:56:34.606980 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:34.606948 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b3bf0e-6xhtq" event={"ID":"6cdbecf8-2c06-48d9-8f80-9ca469289354","Type":"ContainerDied","Data":"b4c29fbfdc6cdebcabdb5f50ae20cb30cbe44d338a1944cca3505047f192e0a4"} Nov 03 20:56:34.607116 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:34.606958 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b3bf0e-6xhtq" Nov 03 20:56:34.607116 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:34.607006 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4c29fbfdc6cdebcabdb5f50ae20cb30cbe44d338a1944cca3505047f192e0a4" Nov 03 20:56:35.516338 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:35.516308 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc"] Nov 03 20:56:35.516894 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:35.516880 4305 cpu_manager.go:414] "containerMap: removing container" podUID="6cdbecf8-2c06-48d9-8f80-9ca469289354" containerName="revoke-keys" Nov 03 20:56:35.516939 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:35.516896 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cdbecf8-2c06-48d9-8f80-9ca469289354" containerName="revoke-keys" Nov 03 20:56:35.516939 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:35.516909 4305 cpu_manager.go:414] "containerMap: removing container" podUID="97f3c196-0376-41ae-8b28-930d728277dd" containerName="revoke-keys" Nov 03 20:56:35.516939 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:35.516914 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97f3c196-0376-41ae-8b28-930d728277dd" containerName="revoke-keys" Nov 03 20:56:35.517028 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:35.516989 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cdbecf8-2c06-48d9-8f80-9ca469289354" containerName="revoke-keys" Nov 03 20:56:35.517028 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:35.516997 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="97f3c196-0376-41ae-8b28-930d728277dd" containerName="revoke-keys" Nov 03 20:56:35.520506 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:35.520488 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc" Nov 03 20:56:35.603480 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:35.602757 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rgx2\" (UniqueName: \"kubernetes.io/projected/6a908050-5ca3-4977-b279-67965ac30de1-kube-api-access-2rgx2\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc\" (UID: \"6a908050-5ca3-4977-b279-67965ac30de1\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc" Nov 03 20:56:35.603480 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:35.602828 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a908050-5ca3-4977-b279-67965ac30de1-istio-envoy\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc\" (UID: \"6a908050-5ca3-4977-b279-67965ac30de1\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc" Nov 03 20:56:35.605651 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:35.605595 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a908050-5ca3-4977-b279-67965ac30de1-workload-socket\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc\" (UID: \"6a908050-5ca3-4977-b279-67965ac30de1\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc" Nov 03 20:56:35.605925 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:35.605893 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a908050-5ca3-4977-b279-67965ac30de1-credential-socket\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc\" (UID: \"6a908050-5ca3-4977-b279-67965ac30de1\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc" Nov 03 20:56:35.606043 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:35.605967 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a908050-5ca3-4977-b279-67965ac30de1-istio-data\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc\" (UID: \"6a908050-5ca3-4977-b279-67965ac30de1\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc" Nov 03 20:56:35.606123 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:35.606049 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a908050-5ca3-4977-b279-67965ac30de1-istiod-ca-cert\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc\" (UID: \"6a908050-5ca3-4977-b279-67965ac30de1\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc" Nov 03 20:56:35.606123 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:35.606096 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a908050-5ca3-4977-b279-67965ac30de1-workload-certs\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc\" (UID: \"6a908050-5ca3-4977-b279-67965ac30de1\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc" Nov 03 20:56:35.606230 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:35.606125 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a908050-5ca3-4977-b279-67965ac30de1-istio-token\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc\" (UID: \"6a908050-5ca3-4977-b279-67965ac30de1\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc" Nov 03 20:56:35.606230 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:35.606164 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a908050-5ca3-4977-b279-67965ac30de1-istio-podinfo\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc\" (UID: \"6a908050-5ca3-4977-b279-67965ac30de1\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc" Nov 03 20:56:35.609595 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:35.609568 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc"] Nov 03 20:56:35.707550 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:35.707501 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a908050-5ca3-4977-b279-67965ac30de1-workload-socket\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc\" (UID: \"6a908050-5ca3-4977-b279-67965ac30de1\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc" Nov 03 20:56:35.707748 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:35.707557 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a908050-5ca3-4977-b279-67965ac30de1-credential-socket\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc\" (UID: \"6a908050-5ca3-4977-b279-67965ac30de1\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc" Nov 03 20:56:35.707748 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:35.707607 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a908050-5ca3-4977-b279-67965ac30de1-istio-data\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc\" (UID: \"6a908050-5ca3-4977-b279-67965ac30de1\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc" Nov 03 20:56:35.707748 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:35.707673 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a908050-5ca3-4977-b279-67965ac30de1-istiod-ca-cert\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc\" (UID: \"6a908050-5ca3-4977-b279-67965ac30de1\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc" Nov 03 20:56:35.707748 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:35.707710 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a908050-5ca3-4977-b279-67965ac30de1-workload-certs\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc\" (UID: \"6a908050-5ca3-4977-b279-67965ac30de1\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc" Nov 03 20:56:35.707748 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:35.707736 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a908050-5ca3-4977-b279-67965ac30de1-istio-token\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc\" (UID: \"6a908050-5ca3-4977-b279-67965ac30de1\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc" Nov 03 20:56:35.708011 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:35.707775 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a908050-5ca3-4977-b279-67965ac30de1-istio-podinfo\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc\" (UID: \"6a908050-5ca3-4977-b279-67965ac30de1\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc" Nov 03 20:56:35.708011 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:35.707867 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rgx2\" (UniqueName: \"kubernetes.io/projected/6a908050-5ca3-4977-b279-67965ac30de1-kube-api-access-2rgx2\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc\" (UID: \"6a908050-5ca3-4977-b279-67965ac30de1\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc" Nov 03 20:56:35.708100 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:35.708005 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a908050-5ca3-4977-b279-67965ac30de1-workload-socket\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc\" (UID: \"6a908050-5ca3-4977-b279-67965ac30de1\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc" Nov 03 20:56:35.708100 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:35.708041 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a908050-5ca3-4977-b279-67965ac30de1-credential-socket\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc\" (UID: \"6a908050-5ca3-4977-b279-67965ac30de1\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc" Nov 03 20:56:35.708100 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:35.708073 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a908050-5ca3-4977-b279-67965ac30de1-istio-envoy\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc\" (UID: \"6a908050-5ca3-4977-b279-67965ac30de1\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc" Nov 03 20:56:35.708252 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:35.708103 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a908050-5ca3-4977-b279-67965ac30de1-istio-data\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc\" (UID: \"6a908050-5ca3-4977-b279-67965ac30de1\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc" Nov 03 20:56:35.708252 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:35.708201 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a908050-5ca3-4977-b279-67965ac30de1-workload-certs\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc\" (UID: \"6a908050-5ca3-4977-b279-67965ac30de1\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc" Nov 03 20:56:35.708427 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:35.708406 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a908050-5ca3-4977-b279-67965ac30de1-istiod-ca-cert\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc\" (UID: \"6a908050-5ca3-4977-b279-67965ac30de1\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc" Nov 03 20:56:35.710546 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:35.710521 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a908050-5ca3-4977-b279-67965ac30de1-istio-envoy\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc\" (UID: \"6a908050-5ca3-4977-b279-67965ac30de1\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc" Nov 03 20:56:35.710675 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:35.710647 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a908050-5ca3-4977-b279-67965ac30de1-istio-podinfo\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc\" (UID: \"6a908050-5ca3-4977-b279-67965ac30de1\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc" Nov 03 20:56:35.768641 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:35.768576 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a908050-5ca3-4977-b279-67965ac30de1-istio-token\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc\" (UID: \"6a908050-5ca3-4977-b279-67965ac30de1\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc" Nov 03 20:56:35.769063 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:35.769036 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rgx2\" (UniqueName: \"kubernetes.io/projected/6a908050-5ca3-4977-b279-67965ac30de1-kube-api-access-2rgx2\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc\" (UID: \"6a908050-5ca3-4977-b279-67965ac30de1\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc" Nov 03 20:56:35.835135 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:35.835097 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc" Nov 03 20:56:36.268356 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:56:36.268327 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a908050_5ca3_4977_b279_67965ac30de1.slice/crio-5af04a7685814a46e482885deb0a3fd73afb8f8f9714114124f737fd42cf996f WatchSource:0}: Error finding container 5af04a7685814a46e482885deb0a3fd73afb8f8f9714114124f737fd42cf996f: Status 404 returned error can't find the container with id 5af04a7685814a46e482885deb0a3fd73afb8f8f9714114124f737fd42cf996f Nov 03 20:56:36.268518 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:36.268474 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc"] Nov 03 20:56:36.270999 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:36.270967 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:56:36.271080 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:36.271031 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:56:36.271080 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:36.271060 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:56:36.334049 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:36.334021 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" Nov 03 20:56:36.347330 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:36.347298 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" Nov 03 20:56:36.625744 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:36.625655 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc" event={"ID":"6a908050-5ca3-4977-b279-67965ac30de1","Type":"ContainerStarted","Data":"fdddee4a24fda9da49e1e23c26953af05b709276af091e6cf4da8eb05fe626b9"} Nov 03 20:56:36.625744 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:36.625709 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc" event={"ID":"6a908050-5ca3-4977-b279-67965ac30de1","Type":"ContainerStarted","Data":"5af04a7685814a46e482885deb0a3fd73afb8f8f9714114124f737fd42cf996f"} Nov 03 20:56:36.640905 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:36.640872 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" Nov 03 20:56:36.749939 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:36.749870 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc" podStartSLOduration=1.7498512590000002 podStartE2EDuration="1.749851259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:56:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:56:36.745341665 +0000 UTC m=+1031.970661456" watchObservedRunningTime="2026-11-03 20:56:36.749851259 +0000 UTC m=+1031.975171050" Nov 03 20:56:36.836110 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:36.836072 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc" Nov 03 20:56:37.507044 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:37.507008 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc281286-gk97p"] Nov 03 20:56:37.511250 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:37.511231 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc281286-gk97p" Nov 03 20:56:37.617463 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:37.617403 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:56:37.617674 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:37.617590 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77vtd\"" Nov 03 20:56:37.617674 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:37.617655 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:56:37.618059 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:37.618041 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:56:37.623355 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:37.623323 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc281286-gk97p"] Nov 03 20:56:37.632321 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:37.632289 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d986989-2d1b-428a-b124-ecbce2ec3943-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc281286-gk97p\" (UID: \"0d986989-2d1b-428a-b124-ecbce2ec3943\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc281286-gk97p" Nov 03 20:56:37.632855 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:37.632451 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnsnm\" (UniqueName: \"kubernetes.io/projected/0d986989-2d1b-428a-b124-ecbce2ec3943-kube-api-access-tnsnm\") pod \"maas-api-revoke-sub-fc281286-gk97p\" (UID: \"0d986989-2d1b-428a-b124-ecbce2ec3943\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc281286-gk97p" Nov 03 20:56:37.632855 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:37.632567 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d986989-2d1b-428a-b124-ecbce2ec3943-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc281286-gk97p\" (UID: \"0d986989-2d1b-428a-b124-ecbce2ec3943\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc281286-gk97p" Nov 03 20:56:37.733924 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:37.733881 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d986989-2d1b-428a-b124-ecbce2ec3943-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc281286-gk97p\" (UID: \"0d986989-2d1b-428a-b124-ecbce2ec3943\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc281286-gk97p" Nov 03 20:56:37.734184 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:37.734159 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnsnm\" (UniqueName: \"kubernetes.io/projected/0d986989-2d1b-428a-b124-ecbce2ec3943-kube-api-access-tnsnm\") pod \"maas-api-revoke-sub-fc281286-gk97p\" (UID: \"0d986989-2d1b-428a-b124-ecbce2ec3943\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc281286-gk97p" Nov 03 20:56:37.734347 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:37.734322 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d986989-2d1b-428a-b124-ecbce2ec3943-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc281286-gk97p\" (UID: \"0d986989-2d1b-428a-b124-ecbce2ec3943\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc281286-gk97p" Nov 03 20:56:37.734694 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:37.734676 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d986989-2d1b-428a-b124-ecbce2ec3943-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc281286-gk97p\" (UID: \"0d986989-2d1b-428a-b124-ecbce2ec3943\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc281286-gk97p" Nov 03 20:56:37.736869 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:37.736852 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d986989-2d1b-428a-b124-ecbce2ec3943-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc281286-gk97p\" (UID: \"0d986989-2d1b-428a-b124-ecbce2ec3943\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc281286-gk97p" Nov 03 20:56:37.795229 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:37.795154 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnsnm\" (UniqueName: \"kubernetes.io/projected/0d986989-2d1b-428a-b124-ecbce2ec3943-kube-api-access-tnsnm\") pod \"maas-api-revoke-sub-fc281286-gk97p\" (UID: \"0d986989-2d1b-428a-b124-ecbce2ec3943\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc281286-gk97p" Nov 03 20:56:37.825856 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:37.825823 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc281286-gk97p" Nov 03 20:56:37.837237 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:37.837209 4305 patch_prober.go:29] interesting pod/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 20:56:37.837377 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:37.837262 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc" podUID="6a908050-5ca3-4977-b279-67965ac30de1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 20:56:38.232405 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:38.232378 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc281286-gk97p"] Nov 03 20:56:38.233414 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:56:38.233386 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d986989_2d1b_428a_b124_ecbce2ec3943.slice/crio-cf480194cfdf1b3c7ba31e35424e93e1f5035fe6feff062989a0a44a0853c96d WatchSource:0}: Error finding container cf480194cfdf1b3c7ba31e35424e93e1f5035fe6feff062989a0a44a0853c96d: Status 404 returned error can't find the container with id cf480194cfdf1b3c7ba31e35424e93e1f5035fe6feff062989a0a44a0853c96d Nov 03 20:56:38.643028 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:38.642995 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc281286-gk97p" event={"ID":"0d986989-2d1b-428a-b124-ecbce2ec3943","Type":"ContainerStarted","Data":"cf480194cfdf1b3c7ba31e35424e93e1f5035fe6feff062989a0a44a0853c96d"} Nov 03 20:56:38.835609 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:38.835522 4305 patch_prober.go:29] interesting pod/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 20:56:38.835609 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:38.835582 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc" podUID="6a908050-5ca3-4977-b279-67965ac30de1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 20:56:39.673459 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:39.651623 4305 generic.go:356] "Generic (PLEG): container finished" podID="0d986989-2d1b-428a-b124-ecbce2ec3943" containerID="f666760b40b1db44c3ca5931742d2edab5a22e3c462266254e313e4f7181c8dd" exitCode=0 Nov 03 20:56:39.673459 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:39.651679 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc281286-gk97p" event={"ID":"0d986989-2d1b-428a-b124-ecbce2ec3943","Type":"ContainerDied","Data":"f666760b40b1db44c3ca5931742d2edab5a22e3c462266254e313e4f7181c8dd"} Nov 03 20:56:39.836266 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:39.836231 4305 patch_prober.go:29] interesting pod/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 20:56:39.836427 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:39.836282 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc" podUID="6a908050-5ca3-4977-b279-67965ac30de1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 20:56:40.836223 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:40.836192 4305 patch_prober.go:29] interesting pod/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 20:56:40.836696 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:40.836245 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc" podUID="6a908050-5ca3-4977-b279-67965ac30de1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 20:56:40.893527 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:40.893494 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc281286-gk97p" Nov 03 20:56:40.993812 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:40.993718 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d986989-2d1b-428a-b124-ecbce2ec3943-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d986989-2d1b-428a-b124-ecbce2ec3943-maas-api-openshift-ca\") pod \"0d986989-2d1b-428a-b124-ecbce2ec3943\" (UID: \"0d986989-2d1b-428a-b124-ecbce2ec3943\") " Nov 03 20:56:40.993812 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:40.993784 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d986989-2d1b-428a-b124-ecbce2ec3943-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d986989-2d1b-428a-b124-ecbce2ec3943-maas-api-xks-ca\") pod \"0d986989-2d1b-428a-b124-ecbce2ec3943\" (UID: \"0d986989-2d1b-428a-b124-ecbce2ec3943\") " Nov 03 20:56:40.994060 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:40.994004 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d986989-2d1b-428a-b124-ecbce2ec3943-kube-api-access-tnsnm\" (UniqueName: \"kubernetes.io/projected/0d986989-2d1b-428a-b124-ecbce2ec3943-kube-api-access-tnsnm\") pod \"0d986989-2d1b-428a-b124-ecbce2ec3943\" (UID: \"0d986989-2d1b-428a-b124-ecbce2ec3943\") " Nov 03 20:56:40.994164 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:40.994140 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d986989-2d1b-428a-b124-ecbce2ec3943-maas-api-openshift-ca" pod "0d986989-2d1b-428a-b124-ecbce2ec3943" (UID: "0d986989-2d1b-428a-b124-ecbce2ec3943"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:56:40.994563 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:40.994540 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d986989-2d1b-428a-b124-ecbce2ec3943-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:56:40.996271 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:40.996250 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d986989-2d1b-428a-b124-ecbce2ec3943-maas-api-xks-ca" pod "0d986989-2d1b-428a-b124-ecbce2ec3943" (UID: "0d986989-2d1b-428a-b124-ecbce2ec3943"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:56:40.996367 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:40.996304 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d986989-2d1b-428a-b124-ecbce2ec3943-kube-api-access-tnsnm" pod "0d986989-2d1b-428a-b124-ecbce2ec3943" (UID: "0d986989-2d1b-428a-b124-ecbce2ec3943"). InnerVolumeSpecName "kube-api-access-tnsnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:56:41.096083 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:41.096043 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d986989-2d1b-428a-b124-ecbce2ec3943-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:56:41.096083 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:41.096075 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnsnm\" (UniqueName: \"kubernetes.io/projected/0d986989-2d1b-428a-b124-ecbce2ec3943-kube-api-access-tnsnm\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:56:41.690060 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:41.690027 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc281286-gk97p" event={"ID":"0d986989-2d1b-428a-b124-ecbce2ec3943","Type":"ContainerDied","Data":"cf480194cfdf1b3c7ba31e35424e93e1f5035fe6feff062989a0a44a0853c96d"} Nov 03 20:56:41.690060 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:41.690055 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc281286-gk97p" Nov 03 20:56:41.690330 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:41.690082 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf480194cfdf1b3c7ba31e35424e93e1f5035fe6feff062989a0a44a0853c96d" Nov 03 20:56:41.835780 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:41.835741 4305 patch_prober.go:29] interesting pod/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 20:56:41.835969 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:41.835812 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc" podUID="6a908050-5ca3-4977-b279-67965ac30de1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 20:56:42.836321 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:42.836282 4305 patch_prober.go:29] interesting pod/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 03 20:56:42.836760 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:42.836351 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc" podUID="6a908050-5ca3-4977-b279-67965ac30de1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 03 20:56:43.247233 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:43.247200 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 20:56:43.850799 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:43.850765 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc" Nov 03 20:56:43.861521 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:43.860860 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc" Nov 03 20:56:43.868790 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:43.868761 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc" Nov 03 20:56:46.322958 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:56:46.322921 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 20:56:46.612758 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:46.612680 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-053094b3-fmfh2"] Nov 03 20:56:46.613256 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:46.613242 4305 cpu_manager.go:414] "containerMap: removing container" podUID="0d986989-2d1b-428a-b124-ecbce2ec3943" containerName="revoke-keys" Nov 03 20:56:46.613300 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:46.613258 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d986989-2d1b-428a-b124-ecbce2ec3943" containerName="revoke-keys" Nov 03 20:56:46.613350 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:46.613341 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d986989-2d1b-428a-b124-ecbce2ec3943" containerName="revoke-keys" Nov 03 20:56:46.616802 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:46.616786 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-053094b3-fmfh2" Nov 03 20:56:46.647519 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:46.647492 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94566a28-565f-4d08-8178-ba8d3da0b963-maas-api-xks-ca\") pod \"maas-api-revoke-sub-053094b3-fmfh2\" (UID: \"94566a28-565f-4d08-8178-ba8d3da0b963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-053094b3-fmfh2" Nov 03 20:56:46.647667 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:46.647546 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjsls\" (UniqueName: \"kubernetes.io/projected/94566a28-565f-4d08-8178-ba8d3da0b963-kube-api-access-zjsls\") pod \"maas-api-revoke-sub-053094b3-fmfh2\" (UID: \"94566a28-565f-4d08-8178-ba8d3da0b963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-053094b3-fmfh2" Nov 03 20:56:46.647667 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:46.647608 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94566a28-565f-4d08-8178-ba8d3da0b963-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-053094b3-fmfh2\" (UID: \"94566a28-565f-4d08-8178-ba8d3da0b963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-053094b3-fmfh2" Nov 03 20:56:46.670459 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:46.670406 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:56:46.675215 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:46.675183 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-053094b3-fmfh2"] Nov 03 20:56:46.719906 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:46.719863 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:56:46.720504 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:46.720472 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77vtd\"" Nov 03 20:56:46.725659 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:46.725626 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:56:46.748602 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:46.748571 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94566a28-565f-4d08-8178-ba8d3da0b963-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-053094b3-fmfh2\" (UID: \"94566a28-565f-4d08-8178-ba8d3da0b963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-053094b3-fmfh2" Nov 03 20:56:46.748778 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:46.748712 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94566a28-565f-4d08-8178-ba8d3da0b963-maas-api-xks-ca\") pod \"maas-api-revoke-sub-053094b3-fmfh2\" (UID: \"94566a28-565f-4d08-8178-ba8d3da0b963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-053094b3-fmfh2" Nov 03 20:56:46.748778 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:46.748758 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjsls\" (UniqueName: \"kubernetes.io/projected/94566a28-565f-4d08-8178-ba8d3da0b963-kube-api-access-zjsls\") pod \"maas-api-revoke-sub-053094b3-fmfh2\" (UID: \"94566a28-565f-4d08-8178-ba8d3da0b963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-053094b3-fmfh2" Nov 03 20:56:46.749288 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:46.749263 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94566a28-565f-4d08-8178-ba8d3da0b963-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-053094b3-fmfh2\" (UID: \"94566a28-565f-4d08-8178-ba8d3da0b963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-053094b3-fmfh2" Nov 03 20:56:46.751105 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:46.751080 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94566a28-565f-4d08-8178-ba8d3da0b963-maas-api-xks-ca\") pod \"maas-api-revoke-sub-053094b3-fmfh2\" (UID: \"94566a28-565f-4d08-8178-ba8d3da0b963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-053094b3-fmfh2" Nov 03 20:56:46.810778 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:46.810747 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjsls\" (UniqueName: \"kubernetes.io/projected/94566a28-565f-4d08-8178-ba8d3da0b963-kube-api-access-zjsls\") pod \"maas-api-revoke-sub-053094b3-fmfh2\" (UID: \"94566a28-565f-4d08-8178-ba8d3da0b963\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-053094b3-fmfh2" Nov 03 20:56:46.931079 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:46.931046 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-053094b3-fmfh2" Nov 03 20:56:46.941412 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:46.941383 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38f52da4-r4b2r"] Nov 03 20:56:46.946356 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:46.946338 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38f52da4-r4b2r" Nov 03 20:56:47.019271 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:47.019236 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38f52da4-r4b2r"] Nov 03 20:56:47.051263 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:47.051232 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7e2ad8b-2ee3-4fb4-8850-c6bf7d0f85e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38f52da4-r4b2r\" (UID: \"d7e2ad8b-2ee3-4fb4-8850-c6bf7d0f85e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38f52da4-r4b2r" Nov 03 20:56:47.051263 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:47.051268 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7e2ad8b-2ee3-4fb4-8850-c6bf7d0f85e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38f52da4-r4b2r\" (UID: \"d7e2ad8b-2ee3-4fb4-8850-c6bf7d0f85e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38f52da4-r4b2r" Nov 03 20:56:47.051550 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:47.051412 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv5mc\" (UniqueName: \"kubernetes.io/projected/d7e2ad8b-2ee3-4fb4-8850-c6bf7d0f85e8-kube-api-access-tv5mc\") pod \"maas-api-revoke-sub-38f52da4-r4b2r\" (UID: \"d7e2ad8b-2ee3-4fb4-8850-c6bf7d0f85e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38f52da4-r4b2r" Nov 03 20:56:47.152842 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:47.152802 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7e2ad8b-2ee3-4fb4-8850-c6bf7d0f85e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38f52da4-r4b2r\" (UID: \"d7e2ad8b-2ee3-4fb4-8850-c6bf7d0f85e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38f52da4-r4b2r" Nov 03 20:56:47.152842 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:47.152842 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7e2ad8b-2ee3-4fb4-8850-c6bf7d0f85e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38f52da4-r4b2r\" (UID: \"d7e2ad8b-2ee3-4fb4-8850-c6bf7d0f85e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38f52da4-r4b2r" Nov 03 20:56:47.153051 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:47.152976 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv5mc\" (UniqueName: \"kubernetes.io/projected/d7e2ad8b-2ee3-4fb4-8850-c6bf7d0f85e8-kube-api-access-tv5mc\") pod \"maas-api-revoke-sub-38f52da4-r4b2r\" (UID: \"d7e2ad8b-2ee3-4fb4-8850-c6bf7d0f85e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38f52da4-r4b2r" Nov 03 20:56:47.153562 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:47.153542 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7e2ad8b-2ee3-4fb4-8850-c6bf7d0f85e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38f52da4-r4b2r\" (UID: \"d7e2ad8b-2ee3-4fb4-8850-c6bf7d0f85e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38f52da4-r4b2r" Nov 03 20:56:47.155428 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:47.155404 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7e2ad8b-2ee3-4fb4-8850-c6bf7d0f85e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38f52da4-r4b2r\" (UID: \"d7e2ad8b-2ee3-4fb4-8850-c6bf7d0f85e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38f52da4-r4b2r" Nov 03 20:56:47.213986 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:47.213915 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv5mc\" (UniqueName: \"kubernetes.io/projected/d7e2ad8b-2ee3-4fb4-8850-c6bf7d0f85e8-kube-api-access-tv5mc\") pod \"maas-api-revoke-sub-38f52da4-r4b2r\" (UID: \"d7e2ad8b-2ee3-4fb4-8850-c6bf7d0f85e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38f52da4-r4b2r" Nov 03 20:56:47.277688 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:47.277651 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38f52da4-r4b2r" Nov 03 20:56:47.314320 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:47.314293 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-053094b3-fmfh2"] Nov 03 20:56:47.315882 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:56:47.315840 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94566a28_565f_4d08_8178_ba8d3da0b963.slice/crio-cff330a5eda93db6814479df5d3e85c17d26051afde2938dab50c199f0bcf5a3 WatchSource:0}: Error finding container cff330a5eda93db6814479df5d3e85c17d26051afde2938dab50c199f0bcf5a3: Status 404 returned error can't find the container with id cff330a5eda93db6814479df5d3e85c17d26051afde2938dab50c199f0bcf5a3 Nov 03 20:56:47.658450 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:47.658407 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38f52da4-r4b2r"] Nov 03 20:56:47.659058 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:56:47.659034 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7e2ad8b_2ee3_4fb4_8850_c6bf7d0f85e8.slice/crio-474b5db6b4b182822622290b89406dc537be7b5ef548ba6576378b1223988f55 WatchSource:0}: Error finding container 474b5db6b4b182822622290b89406dc537be7b5ef548ba6576378b1223988f55: Status 404 returned error can't find the container with id 474b5db6b4b182822622290b89406dc537be7b5ef548ba6576378b1223988f55 Nov 03 20:56:47.755474 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:47.755421 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38f52da4-r4b2r" event={"ID":"d7e2ad8b-2ee3-4fb4-8850-c6bf7d0f85e8","Type":"ContainerStarted","Data":"474b5db6b4b182822622290b89406dc537be7b5ef548ba6576378b1223988f55"} Nov 03 20:56:47.757101 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:47.757073 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-053094b3-fmfh2" event={"ID":"94566a28-565f-4d08-8178-ba8d3da0b963","Type":"ContainerStarted","Data":"cff330a5eda93db6814479df5d3e85c17d26051afde2938dab50c199f0bcf5a3"} Nov 03 20:56:48.764841 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:48.764811 4305 generic.go:356] "Generic (PLEG): container finished" podID="d7e2ad8b-2ee3-4fb4-8850-c6bf7d0f85e8" containerID="815381d59aab0887eb76d120a2f5d5d78c67c6b3fd29a590c66220fff2b0e9bb" exitCode=0 Nov 03 20:56:48.765263 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:48.764898 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38f52da4-r4b2r" event={"ID":"d7e2ad8b-2ee3-4fb4-8850-c6bf7d0f85e8","Type":"ContainerDied","Data":"815381d59aab0887eb76d120a2f5d5d78c67c6b3fd29a590c66220fff2b0e9bb"} Nov 03 20:56:48.766558 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:48.766534 4305 generic.go:356] "Generic (PLEG): container finished" podID="94566a28-565f-4d08-8178-ba8d3da0b963" containerID="e094719213d34a0e3d223891432f1a86cc1bbfdea6c2401700a41d3279e4213b" exitCode=0 Nov 03 20:56:48.766681 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:48.766611 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-053094b3-fmfh2" event={"ID":"94566a28-565f-4d08-8178-ba8d3da0b963","Type":"ContainerDied","Data":"e094719213d34a0e3d223891432f1a86cc1bbfdea6c2401700a41d3279e4213b"} Nov 03 20:56:49.972601 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:49.972548 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38f52da4-r4b2r" Nov 03 20:56:49.976200 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:49.976172 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-053094b3-fmfh2" Nov 03 20:56:50.090708 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:50.090621 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7e2ad8b-2ee3-4fb4-8850-c6bf7d0f85e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7e2ad8b-2ee3-4fb4-8850-c6bf7d0f85e8-maas-api-xks-ca\") pod \"d7e2ad8b-2ee3-4fb4-8850-c6bf7d0f85e8\" (UID: \"d7e2ad8b-2ee3-4fb4-8850-c6bf7d0f85e8\") " Nov 03 20:56:50.090881 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:50.090765 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7e2ad8b-2ee3-4fb4-8850-c6bf7d0f85e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7e2ad8b-2ee3-4fb4-8850-c6bf7d0f85e8-maas-api-openshift-ca\") pod \"d7e2ad8b-2ee3-4fb4-8850-c6bf7d0f85e8\" (UID: \"d7e2ad8b-2ee3-4fb4-8850-c6bf7d0f85e8\") " Nov 03 20:56:50.090881 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:50.090792 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7e2ad8b-2ee3-4fb4-8850-c6bf7d0f85e8-kube-api-access-tv5mc\" (UniqueName: \"kubernetes.io/projected/d7e2ad8b-2ee3-4fb4-8850-c6bf7d0f85e8-kube-api-access-tv5mc\") pod \"d7e2ad8b-2ee3-4fb4-8850-c6bf7d0f85e8\" (UID: \"d7e2ad8b-2ee3-4fb4-8850-c6bf7d0f85e8\") " Nov 03 20:56:50.090991 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:50.090890 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94566a28-565f-4d08-8178-ba8d3da0b963-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94566a28-565f-4d08-8178-ba8d3da0b963-maas-api-openshift-ca\") pod \"94566a28-565f-4d08-8178-ba8d3da0b963\" (UID: \"94566a28-565f-4d08-8178-ba8d3da0b963\") " Nov 03 20:56:50.090991 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:50.090945 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94566a28-565f-4d08-8178-ba8d3da0b963-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94566a28-565f-4d08-8178-ba8d3da0b963-maas-api-xks-ca\") pod \"94566a28-565f-4d08-8178-ba8d3da0b963\" (UID: \"94566a28-565f-4d08-8178-ba8d3da0b963\") " Nov 03 20:56:50.090991 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:50.090980 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94566a28-565f-4d08-8178-ba8d3da0b963-kube-api-access-zjsls\" (UniqueName: \"kubernetes.io/projected/94566a28-565f-4d08-8178-ba8d3da0b963-kube-api-access-zjsls\") pod \"94566a28-565f-4d08-8178-ba8d3da0b963\" (UID: \"94566a28-565f-4d08-8178-ba8d3da0b963\") " Nov 03 20:56:50.092016 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:50.091978 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94566a28-565f-4d08-8178-ba8d3da0b963-maas-api-openshift-ca" pod "94566a28-565f-4d08-8178-ba8d3da0b963" (UID: "94566a28-565f-4d08-8178-ba8d3da0b963"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:56:50.093749 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:50.093709 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7e2ad8b-2ee3-4fb4-8850-c6bf7d0f85e8-maas-api-openshift-ca" pod "d7e2ad8b-2ee3-4fb4-8850-c6bf7d0f85e8" (UID: "d7e2ad8b-2ee3-4fb4-8850-c6bf7d0f85e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:56:50.094405 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:50.094376 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94566a28-565f-4d08-8178-ba8d3da0b963-kube-api-access-zjsls" pod "94566a28-565f-4d08-8178-ba8d3da0b963" (UID: "94566a28-565f-4d08-8178-ba8d3da0b963"). InnerVolumeSpecName "kube-api-access-zjsls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:56:50.096013 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:50.095970 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94566a28-565f-4d08-8178-ba8d3da0b963-maas-api-xks-ca" pod "94566a28-565f-4d08-8178-ba8d3da0b963" (UID: "94566a28-565f-4d08-8178-ba8d3da0b963"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:56:50.096143 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:50.096120 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7e2ad8b-2ee3-4fb4-8850-c6bf7d0f85e8-kube-api-access-tv5mc" pod "d7e2ad8b-2ee3-4fb4-8850-c6bf7d0f85e8" (UID: "d7e2ad8b-2ee3-4fb4-8850-c6bf7d0f85e8"). InnerVolumeSpecName "kube-api-access-tv5mc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:56:50.097615 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:50.097581 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7e2ad8b-2ee3-4fb4-8850-c6bf7d0f85e8-maas-api-xks-ca" pod "d7e2ad8b-2ee3-4fb4-8850-c6bf7d0f85e8" (UID: "d7e2ad8b-2ee3-4fb4-8850-c6bf7d0f85e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:56:50.193922 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:50.193887 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7e2ad8b-2ee3-4fb4-8850-c6bf7d0f85e8-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:56:50.193922 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:50.193919 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tv5mc\" (UniqueName: \"kubernetes.io/projected/d7e2ad8b-2ee3-4fb4-8850-c6bf7d0f85e8-kube-api-access-tv5mc\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:56:50.193922 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:50.193929 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94566a28-565f-4d08-8178-ba8d3da0b963-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:56:50.194267 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:50.193939 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94566a28-565f-4d08-8178-ba8d3da0b963-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:56:50.194267 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:50.193950 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjsls\" (UniqueName: \"kubernetes.io/projected/94566a28-565f-4d08-8178-ba8d3da0b963-kube-api-access-zjsls\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:56:50.194267 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:50.193960 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7e2ad8b-2ee3-4fb4-8850-c6bf7d0f85e8-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:56:50.784483 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:50.784420 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38f52da4-r4b2r" Nov 03 20:56:50.784701 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:50.784480 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38f52da4-r4b2r" event={"ID":"d7e2ad8b-2ee3-4fb4-8850-c6bf7d0f85e8","Type":"ContainerDied","Data":"474b5db6b4b182822622290b89406dc537be7b5ef548ba6576378b1223988f55"} Nov 03 20:56:50.784701 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:50.784560 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="474b5db6b4b182822622290b89406dc537be7b5ef548ba6576378b1223988f55" Nov 03 20:56:50.785964 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:50.785943 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-053094b3-fmfh2" Nov 03 20:56:50.785964 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:50.785954 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-053094b3-fmfh2" event={"ID":"94566a28-565f-4d08-8178-ba8d3da0b963","Type":"ContainerDied","Data":"cff330a5eda93db6814479df5d3e85c17d26051afde2938dab50c199f0bcf5a3"} Nov 03 20:56:50.786188 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:50.785997 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cff330a5eda93db6814479df5d3e85c17d26051afde2938dab50c199f0bcf5a3" Nov 03 20:56:52.379651 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:52.379618 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6532ca59-p7h2d"] Nov 03 20:56:52.380189 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:52.380172 4305 cpu_manager.go:414] "containerMap: removing container" podUID="d7e2ad8b-2ee3-4fb4-8850-c6bf7d0f85e8" containerName="revoke-keys" Nov 03 20:56:52.380189 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:52.380190 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7e2ad8b-2ee3-4fb4-8850-c6bf7d0f85e8" containerName="revoke-keys" Nov 03 20:56:52.380339 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:52.380213 4305 cpu_manager.go:414] "containerMap: removing container" podUID="94566a28-565f-4d08-8178-ba8d3da0b963" containerName="revoke-keys" Nov 03 20:56:52.380339 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:52.380218 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94566a28-565f-4d08-8178-ba8d3da0b963" containerName="revoke-keys" Nov 03 20:56:52.380339 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:52.380317 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7e2ad8b-2ee3-4fb4-8850-c6bf7d0f85e8" containerName="revoke-keys" Nov 03 20:56:52.380339 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:52.380333 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="94566a28-565f-4d08-8178-ba8d3da0b963" containerName="revoke-keys" Nov 03 20:56:52.384079 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:52.384061 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6532ca59-p7h2d" Nov 03 20:56:52.421526 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:52.421490 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bef6538-68a6-4d99-baa9-a7bf16dcb149-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6532ca59-p7h2d\" (UID: \"1bef6538-68a6-4d99-baa9-a7bf16dcb149\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6532ca59-p7h2d" Nov 03 20:56:52.421714 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:52.421692 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5tkw\" (UniqueName: \"kubernetes.io/projected/1bef6538-68a6-4d99-baa9-a7bf16dcb149-kube-api-access-m5tkw\") pod \"maas-api-revoke-sub-6532ca59-p7h2d\" (UID: \"1bef6538-68a6-4d99-baa9-a7bf16dcb149\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6532ca59-p7h2d" Nov 03 20:56:52.421763 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:52.421750 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bef6538-68a6-4d99-baa9-a7bf16dcb149-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6532ca59-p7h2d\" (UID: \"1bef6538-68a6-4d99-baa9-a7bf16dcb149\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6532ca59-p7h2d" Nov 03 20:56:52.436640 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:52.436609 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:56:52.486063 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:52.486031 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:56:52.486264 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:52.486111 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:56:52.486264 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:52.486179 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77vtd\"" Nov 03 20:56:52.490142 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:52.490120 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6532ca59-p7h2d"] Nov 03 20:56:52.523594 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:52.523553 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bef6538-68a6-4d99-baa9-a7bf16dcb149-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6532ca59-p7h2d\" (UID: \"1bef6538-68a6-4d99-baa9-a7bf16dcb149\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6532ca59-p7h2d" Nov 03 20:56:52.523799 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:52.523674 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bef6538-68a6-4d99-baa9-a7bf16dcb149-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6532ca59-p7h2d\" (UID: \"1bef6538-68a6-4d99-baa9-a7bf16dcb149\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6532ca59-p7h2d" Nov 03 20:56:52.523885 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:52.523829 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5tkw\" (UniqueName: \"kubernetes.io/projected/1bef6538-68a6-4d99-baa9-a7bf16dcb149-kube-api-access-m5tkw\") pod \"maas-api-revoke-sub-6532ca59-p7h2d\" (UID: \"1bef6538-68a6-4d99-baa9-a7bf16dcb149\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6532ca59-p7h2d" Nov 03 20:56:52.524309 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:52.524287 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bef6538-68a6-4d99-baa9-a7bf16dcb149-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6532ca59-p7h2d\" (UID: \"1bef6538-68a6-4d99-baa9-a7bf16dcb149\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6532ca59-p7h2d" Nov 03 20:56:52.526216 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:52.526193 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bef6538-68a6-4d99-baa9-a7bf16dcb149-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6532ca59-p7h2d\" (UID: \"1bef6538-68a6-4d99-baa9-a7bf16dcb149\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6532ca59-p7h2d" Nov 03 20:56:52.586946 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:52.586913 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5tkw\" (UniqueName: \"kubernetes.io/projected/1bef6538-68a6-4d99-baa9-a7bf16dcb149-kube-api-access-m5tkw\") pod \"maas-api-revoke-sub-6532ca59-p7h2d\" (UID: \"1bef6538-68a6-4d99-baa9-a7bf16dcb149\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6532ca59-p7h2d" Nov 03 20:56:52.697980 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:52.697947 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6532ca59-p7h2d" Nov 03 20:56:53.084848 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:56:53.084802 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bef6538_68a6_4d99_baa9_a7bf16dcb149.slice/crio-9d8c204e6be2408250e92e751b226155aa7c3c9cdc58b2209533b124821b756f WatchSource:0}: Error finding container 9d8c204e6be2408250e92e751b226155aa7c3c9cdc58b2209533b124821b756f: Status 404 returned error can't find the container with id 9d8c204e6be2408250e92e751b226155aa7c3c9cdc58b2209533b124821b756f Nov 03 20:56:53.085350 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:53.085325 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6532ca59-p7h2d"] Nov 03 20:56:53.809402 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:53.809369 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6532ca59-p7h2d" event={"ID":"1bef6538-68a6-4d99-baa9-a7bf16dcb149","Type":"ContainerStarted","Data":"57dcd8efd6ddc3e473dcca0aab985606acb89473104660948b1f1f19cea2900e"} Nov 03 20:56:53.809928 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:53.809421 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6532ca59-p7h2d" event={"ID":"1bef6538-68a6-4d99-baa9-a7bf16dcb149","Type":"ContainerStarted","Data":"9d8c204e6be2408250e92e751b226155aa7c3c9cdc58b2209533b124821b756f"} Nov 03 20:56:53.926139 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:53.926081 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6532ca59-p7h2d" podStartSLOduration=1.926067807 podStartE2EDuration="1.926067807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:56:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:56:53.92458219 +0000 UTC m=+1049.149901982" watchObservedRunningTime="2026-11-03 20:56:53.926067807 +0000 UTC m=+1049.151387599" Nov 03 20:56:54.818799 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:54.818765 4305 generic.go:356] "Generic (PLEG): container finished" podID="1bef6538-68a6-4d99-baa9-a7bf16dcb149" containerID="57dcd8efd6ddc3e473dcca0aab985606acb89473104660948b1f1f19cea2900e" exitCode=0 Nov 03 20:56:54.819270 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:54.818819 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6532ca59-p7h2d" event={"ID":"1bef6538-68a6-4d99-baa9-a7bf16dcb149","Type":"ContainerDied","Data":"57dcd8efd6ddc3e473dcca0aab985606acb89473104660948b1f1f19cea2900e"} Nov 03 20:56:55.989050 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:55.989020 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6532ca59-p7h2d" Nov 03 20:56:56.068775 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:56.068741 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bef6538-68a6-4d99-baa9-a7bf16dcb149-kube-api-access-m5tkw\" (UniqueName: \"kubernetes.io/projected/1bef6538-68a6-4d99-baa9-a7bf16dcb149-kube-api-access-m5tkw\") pod \"1bef6538-68a6-4d99-baa9-a7bf16dcb149\" (UID: \"1bef6538-68a6-4d99-baa9-a7bf16dcb149\") " Nov 03 20:56:56.068968 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:56.068802 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bef6538-68a6-4d99-baa9-a7bf16dcb149-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bef6538-68a6-4d99-baa9-a7bf16dcb149-maas-api-xks-ca\") pod \"1bef6538-68a6-4d99-baa9-a7bf16dcb149\" (UID: \"1bef6538-68a6-4d99-baa9-a7bf16dcb149\") " Nov 03 20:56:56.068968 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:56.068952 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bef6538-68a6-4d99-baa9-a7bf16dcb149-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bef6538-68a6-4d99-baa9-a7bf16dcb149-maas-api-openshift-ca\") pod \"1bef6538-68a6-4d99-baa9-a7bf16dcb149\" (UID: \"1bef6538-68a6-4d99-baa9-a7bf16dcb149\") " Nov 03 20:56:56.069258 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:56.069234 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bef6538-68a6-4d99-baa9-a7bf16dcb149-maas-api-openshift-ca" pod "1bef6538-68a6-4d99-baa9-a7bf16dcb149" (UID: "1bef6538-68a6-4d99-baa9-a7bf16dcb149"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:56:56.070953 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:56.070929 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bef6538-68a6-4d99-baa9-a7bf16dcb149-maas-api-xks-ca" pod "1bef6538-68a6-4d99-baa9-a7bf16dcb149" (UID: "1bef6538-68a6-4d99-baa9-a7bf16dcb149"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:56:56.071056 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:56.070999 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bef6538-68a6-4d99-baa9-a7bf16dcb149-kube-api-access-m5tkw" pod "1bef6538-68a6-4d99-baa9-a7bf16dcb149" (UID: "1bef6538-68a6-4d99-baa9-a7bf16dcb149"). InnerVolumeSpecName "kube-api-access-m5tkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:56:56.170956 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:56.170918 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5tkw\" (UniqueName: \"kubernetes.io/projected/1bef6538-68a6-4d99-baa9-a7bf16dcb149-kube-api-access-m5tkw\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:56:56.170956 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:56.170952 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bef6538-68a6-4d99-baa9-a7bf16dcb149-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:56:56.170956 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:56.170964 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bef6538-68a6-4d99-baa9-a7bf16dcb149-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:56:56.832756 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:56.832726 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6532ca59-p7h2d" Nov 03 20:56:56.832756 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:56.832751 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6532ca59-p7h2d" event={"ID":"1bef6538-68a6-4d99-baa9-a7bf16dcb149","Type":"ContainerDied","Data":"9d8c204e6be2408250e92e751b226155aa7c3c9cdc58b2209533b124821b756f"} Nov 03 20:56:56.833000 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:56.832802 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d8c204e6be2408250e92e751b226155aa7c3c9cdc58b2209533b124821b756f" Nov 03 20:56:57.760688 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:57.760656 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9daefaf7-f8gs7"] Nov 03 20:56:57.761310 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:57.761296 4305 cpu_manager.go:414] "containerMap: removing container" podUID="1bef6538-68a6-4d99-baa9-a7bf16dcb149" containerName="revoke-keys" Nov 03 20:56:57.761377 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:57.761317 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bef6538-68a6-4d99-baa9-a7bf16dcb149" containerName="revoke-keys" Nov 03 20:56:57.761418 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:57.761408 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bef6538-68a6-4d99-baa9-a7bf16dcb149" containerName="revoke-keys" Nov 03 20:56:57.765193 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:57.765174 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9daefaf7-f8gs7" Nov 03 20:56:57.817597 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:57.817554 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:56:57.822799 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:57.822768 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9daefaf7-f8gs7"] Nov 03 20:56:57.867083 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:57.867048 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:56:57.867083 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:57.867067 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77vtd\"" Nov 03 20:56:57.867083 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:57.867056 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:56:57.891009 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:57.890970 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsw47\" (UniqueName: \"kubernetes.io/projected/80e4faf1-6b07-4c82-85d5-3807a6573379-kube-api-access-dsw47\") pod \"maas-api-revoke-sub-9daefaf7-f8gs7\" (UID: \"80e4faf1-6b07-4c82-85d5-3807a6573379\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9daefaf7-f8gs7" Nov 03 20:56:57.891225 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:57.891103 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80e4faf1-6b07-4c82-85d5-3807a6573379-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9daefaf7-f8gs7\" (UID: \"80e4faf1-6b07-4c82-85d5-3807a6573379\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9daefaf7-f8gs7" Nov 03 20:56:57.891225 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:57.891186 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80e4faf1-6b07-4c82-85d5-3807a6573379-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9daefaf7-f8gs7\" (UID: \"80e4faf1-6b07-4c82-85d5-3807a6573379\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9daefaf7-f8gs7" Nov 03 20:56:57.992720 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:57.992685 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsw47\" (UniqueName: \"kubernetes.io/projected/80e4faf1-6b07-4c82-85d5-3807a6573379-kube-api-access-dsw47\") pod \"maas-api-revoke-sub-9daefaf7-f8gs7\" (UID: \"80e4faf1-6b07-4c82-85d5-3807a6573379\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9daefaf7-f8gs7" Nov 03 20:56:57.992944 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:57.992921 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80e4faf1-6b07-4c82-85d5-3807a6573379-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9daefaf7-f8gs7\" (UID: \"80e4faf1-6b07-4c82-85d5-3807a6573379\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9daefaf7-f8gs7" Nov 03 20:56:57.993030 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:57.992976 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80e4faf1-6b07-4c82-85d5-3807a6573379-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9daefaf7-f8gs7\" (UID: \"80e4faf1-6b07-4c82-85d5-3807a6573379\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9daefaf7-f8gs7" Nov 03 20:56:57.993774 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:57.993721 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80e4faf1-6b07-4c82-85d5-3807a6573379-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9daefaf7-f8gs7\" (UID: \"80e4faf1-6b07-4c82-85d5-3807a6573379\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9daefaf7-f8gs7" Nov 03 20:56:57.995812 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:57.995787 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80e4faf1-6b07-4c82-85d5-3807a6573379-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9daefaf7-f8gs7\" (UID: \"80e4faf1-6b07-4c82-85d5-3807a6573379\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9daefaf7-f8gs7" Nov 03 20:56:58.053825 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:58.053755 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsw47\" (UniqueName: \"kubernetes.io/projected/80e4faf1-6b07-4c82-85d5-3807a6573379-kube-api-access-dsw47\") pod \"maas-api-revoke-sub-9daefaf7-f8gs7\" (UID: \"80e4faf1-6b07-4c82-85d5-3807a6573379\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9daefaf7-f8gs7" Nov 03 20:56:58.082331 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:58.082303 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9daefaf7-f8gs7" Nov 03 20:56:58.465027 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:58.464996 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9daefaf7-f8gs7"] Nov 03 20:56:58.465486 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:56:58.465406 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80e4faf1_6b07_4c82_85d5_3807a6573379.slice/crio-a19153774e97751e257f70e630861f04ea0265c997b0ed9971982a57301459b2 WatchSource:0}: Error finding container a19153774e97751e257f70e630861f04ea0265c997b0ed9971982a57301459b2: Status 404 returned error can't find the container with id a19153774e97751e257f70e630861f04ea0265c997b0ed9971982a57301459b2 Nov 03 20:56:58.847662 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:58.847615 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9daefaf7-f8gs7" event={"ID":"80e4faf1-6b07-4c82-85d5-3807a6573379","Type":"ContainerStarted","Data":"2a24fb82711a099fdd03fa72b356400287ad62cf8920375622648f1c47c5b731"} Nov 03 20:56:58.848141 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:58.847677 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9daefaf7-f8gs7" event={"ID":"80e4faf1-6b07-4c82-85d5-3807a6573379","Type":"ContainerStarted","Data":"a19153774e97751e257f70e630861f04ea0265c997b0ed9971982a57301459b2"} Nov 03 20:56:58.966420 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:58.966369 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9daefaf7-f8gs7" podStartSLOduration=1.966354532 podStartE2EDuration="1.966354532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:56:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:56:58.961903991 +0000 UTC m=+1054.187223793" watchObservedRunningTime="2026-11-03 20:56:58.966354532 +0000 UTC m=+1054.191674373" Nov 03 20:56:59.855741 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:59.855709 4305 generic.go:356] "Generic (PLEG): container finished" podID="80e4faf1-6b07-4c82-85d5-3807a6573379" containerID="2a24fb82711a099fdd03fa72b356400287ad62cf8920375622648f1c47c5b731" exitCode=0 Nov 03 20:56:59.856153 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:56:59.855791 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9daefaf7-f8gs7" event={"ID":"80e4faf1-6b07-4c82-85d5-3807a6573379","Type":"ContainerDied","Data":"2a24fb82711a099fdd03fa72b356400287ad62cf8920375622648f1c47c5b731"} Nov 03 20:57:01.027628 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:01.027600 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9daefaf7-f8gs7" Nov 03 20:57:01.130918 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:01.130823 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80e4faf1-6b07-4c82-85d5-3807a6573379-kube-api-access-dsw47\" (UniqueName: \"kubernetes.io/projected/80e4faf1-6b07-4c82-85d5-3807a6573379-kube-api-access-dsw47\") pod \"80e4faf1-6b07-4c82-85d5-3807a6573379\" (UID: \"80e4faf1-6b07-4c82-85d5-3807a6573379\") " Nov 03 20:57:01.131099 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:01.130959 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80e4faf1-6b07-4c82-85d5-3807a6573379-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80e4faf1-6b07-4c82-85d5-3807a6573379-maas-api-openshift-ca\") pod \"80e4faf1-6b07-4c82-85d5-3807a6573379\" (UID: \"80e4faf1-6b07-4c82-85d5-3807a6573379\") " Nov 03 20:57:01.131099 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:01.130999 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80e4faf1-6b07-4c82-85d5-3807a6573379-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80e4faf1-6b07-4c82-85d5-3807a6573379-maas-api-xks-ca\") pod \"80e4faf1-6b07-4c82-85d5-3807a6573379\" (UID: \"80e4faf1-6b07-4c82-85d5-3807a6573379\") " Nov 03 20:57:01.131347 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:01.131322 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80e4faf1-6b07-4c82-85d5-3807a6573379-maas-api-openshift-ca" pod "80e4faf1-6b07-4c82-85d5-3807a6573379" (UID: "80e4faf1-6b07-4c82-85d5-3807a6573379"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:57:01.133202 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:01.133180 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80e4faf1-6b07-4c82-85d5-3807a6573379-kube-api-access-dsw47" pod "80e4faf1-6b07-4c82-85d5-3807a6573379" (UID: "80e4faf1-6b07-4c82-85d5-3807a6573379"). InnerVolumeSpecName "kube-api-access-dsw47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:57:01.133264 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:01.133217 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80e4faf1-6b07-4c82-85d5-3807a6573379-maas-api-xks-ca" pod "80e4faf1-6b07-4c82-85d5-3807a6573379" (UID: "80e4faf1-6b07-4c82-85d5-3807a6573379"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:57:01.232210 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:01.232170 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsw47\" (UniqueName: \"kubernetes.io/projected/80e4faf1-6b07-4c82-85d5-3807a6573379-kube-api-access-dsw47\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:57:01.232210 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:01.232204 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80e4faf1-6b07-4c82-85d5-3807a6573379-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:57:01.232210 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:01.232218 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80e4faf1-6b07-4c82-85d5-3807a6573379-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:57:01.873057 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:01.873023 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9daefaf7-f8gs7" Nov 03 20:57:01.873239 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:01.873076 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9daefaf7-f8gs7" event={"ID":"80e4faf1-6b07-4c82-85d5-3807a6573379","Type":"ContainerDied","Data":"a19153774e97751e257f70e630861f04ea0265c997b0ed9971982a57301459b2"} Nov 03 20:57:01.873239 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:01.873120 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a19153774e97751e257f70e630861f04ea0265c997b0ed9971982a57301459b2" Nov 03 20:57:02.265105 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:02.265042 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b5f75e9-7k9qt"] Nov 03 20:57:02.265849 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:02.265829 4305 cpu_manager.go:414] "containerMap: removing container" podUID="80e4faf1-6b07-4c82-85d5-3807a6573379" containerName="revoke-keys" Nov 03 20:57:02.265849 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:02.265850 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80e4faf1-6b07-4c82-85d5-3807a6573379" containerName="revoke-keys" Nov 03 20:57:02.265979 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:02.265934 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="80e4faf1-6b07-4c82-85d5-3807a6573379" containerName="revoke-keys" Nov 03 20:57:02.269576 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:02.269553 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b5f75e9-7k9qt" Nov 03 20:57:02.335337 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:02.335299 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:57:02.335337 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:02.335333 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:57:02.340771 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:02.340742 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:57:02.342620 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:02.342591 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b5f75e9-7k9qt"] Nov 03 20:57:02.354963 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:02.354924 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68525bc3-6784-42a7-9b5b-6c1fe167d454-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b5f75e9-7k9qt\" (UID: \"68525bc3-6784-42a7-9b5b-6c1fe167d454\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b5f75e9-7k9qt" Nov 03 20:57:02.355173 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:02.354981 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blmz5\" (UniqueName: \"kubernetes.io/projected/68525bc3-6784-42a7-9b5b-6c1fe167d454-kube-api-access-blmz5\") pod \"maas-api-revoke-sub-5b5f75e9-7k9qt\" (UID: \"68525bc3-6784-42a7-9b5b-6c1fe167d454\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b5f75e9-7k9qt" Nov 03 20:57:02.355254 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:02.355186 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68525bc3-6784-42a7-9b5b-6c1fe167d454-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b5f75e9-7k9qt\" (UID: \"68525bc3-6784-42a7-9b5b-6c1fe167d454\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b5f75e9-7k9qt" Nov 03 20:57:02.385070 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:02.385034 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77vtd\"" Nov 03 20:57:02.456447 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:02.456399 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68525bc3-6784-42a7-9b5b-6c1fe167d454-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b5f75e9-7k9qt\" (UID: \"68525bc3-6784-42a7-9b5b-6c1fe167d454\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b5f75e9-7k9qt" Nov 03 20:57:02.456636 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:02.456541 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68525bc3-6784-42a7-9b5b-6c1fe167d454-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b5f75e9-7k9qt\" (UID: \"68525bc3-6784-42a7-9b5b-6c1fe167d454\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b5f75e9-7k9qt" Nov 03 20:57:02.456636 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:02.456580 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blmz5\" (UniqueName: \"kubernetes.io/projected/68525bc3-6784-42a7-9b5b-6c1fe167d454-kube-api-access-blmz5\") pod \"maas-api-revoke-sub-5b5f75e9-7k9qt\" (UID: \"68525bc3-6784-42a7-9b5b-6c1fe167d454\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b5f75e9-7k9qt" Nov 03 20:57:02.457281 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:02.457260 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68525bc3-6784-42a7-9b5b-6c1fe167d454-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b5f75e9-7k9qt\" (UID: \"68525bc3-6784-42a7-9b5b-6c1fe167d454\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b5f75e9-7k9qt" Nov 03 20:57:02.459037 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:02.459014 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68525bc3-6784-42a7-9b5b-6c1fe167d454-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b5f75e9-7k9qt\" (UID: \"68525bc3-6784-42a7-9b5b-6c1fe167d454\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b5f75e9-7k9qt" Nov 03 20:57:02.517924 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:02.517844 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blmz5\" (UniqueName: \"kubernetes.io/projected/68525bc3-6784-42a7-9b5b-6c1fe167d454-kube-api-access-blmz5\") pod \"maas-api-revoke-sub-5b5f75e9-7k9qt\" (UID: \"68525bc3-6784-42a7-9b5b-6c1fe167d454\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b5f75e9-7k9qt" Nov 03 20:57:02.593546 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:02.593505 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b5f75e9-7k9qt" Nov 03 20:57:02.980553 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:02.980521 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b5f75e9-7k9qt"] Nov 03 20:57:02.981465 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:57:02.981411 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68525bc3_6784_42a7_9b5b_6c1fe167d454.slice/crio-e539650d65f17fd8d9ff7519bc724510160d7b5505130251af5f6cfd0bf18af8 WatchSource:0}: Error finding container e539650d65f17fd8d9ff7519bc724510160d7b5505130251af5f6cfd0bf18af8: Status 404 returned error can't find the container with id e539650d65f17fd8d9ff7519bc724510160d7b5505130251af5f6cfd0bf18af8 Nov 03 20:57:03.889520 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:03.889485 4305 generic.go:356] "Generic (PLEG): container finished" podID="68525bc3-6784-42a7-9b5b-6c1fe167d454" containerID="9c4ef73d6ff73da4f421f0bff7c7fc214b45f6dff6cb0c5b966d3a38bffb3888" exitCode=0 Nov 03 20:57:03.889520 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:03.889522 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b5f75e9-7k9qt" event={"ID":"68525bc3-6784-42a7-9b5b-6c1fe167d454","Type":"ContainerDied","Data":"9c4ef73d6ff73da4f421f0bff7c7fc214b45f6dff6cb0c5b966d3a38bffb3888"} Nov 03 20:57:03.889948 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:03.889568 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b5f75e9-7k9qt" event={"ID":"68525bc3-6784-42a7-9b5b-6c1fe167d454","Type":"ContainerStarted","Data":"e539650d65f17fd8d9ff7519bc724510160d7b5505130251af5f6cfd0bf18af8"} Nov 03 20:57:05.066070 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:05.066044 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b5f75e9-7k9qt" Nov 03 20:57:05.188677 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:05.188595 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68525bc3-6784-42a7-9b5b-6c1fe167d454-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68525bc3-6784-42a7-9b5b-6c1fe167d454-maas-api-openshift-ca\") pod \"68525bc3-6784-42a7-9b5b-6c1fe167d454\" (UID: \"68525bc3-6784-42a7-9b5b-6c1fe167d454\") " Nov 03 20:57:05.188841 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:05.188738 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68525bc3-6784-42a7-9b5b-6c1fe167d454-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68525bc3-6784-42a7-9b5b-6c1fe167d454-maas-api-xks-ca\") pod \"68525bc3-6784-42a7-9b5b-6c1fe167d454\" (UID: \"68525bc3-6784-42a7-9b5b-6c1fe167d454\") " Nov 03 20:57:05.188916 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:05.188852 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68525bc3-6784-42a7-9b5b-6c1fe167d454-kube-api-access-blmz5\" (UniqueName: \"kubernetes.io/projected/68525bc3-6784-42a7-9b5b-6c1fe167d454-kube-api-access-blmz5\") pod \"68525bc3-6784-42a7-9b5b-6c1fe167d454\" (UID: \"68525bc3-6784-42a7-9b5b-6c1fe167d454\") " Nov 03 20:57:05.189017 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:05.188995 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68525bc3-6784-42a7-9b5b-6c1fe167d454-maas-api-openshift-ca" pod "68525bc3-6784-42a7-9b5b-6c1fe167d454" (UID: "68525bc3-6784-42a7-9b5b-6c1fe167d454"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:57:05.189472 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:05.189402 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68525bc3-6784-42a7-9b5b-6c1fe167d454-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:57:05.191152 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:05.191123 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68525bc3-6784-42a7-9b5b-6c1fe167d454-maas-api-xks-ca" pod "68525bc3-6784-42a7-9b5b-6c1fe167d454" (UID: "68525bc3-6784-42a7-9b5b-6c1fe167d454"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:57:05.191602 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:05.191580 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68525bc3-6784-42a7-9b5b-6c1fe167d454-kube-api-access-blmz5" pod "68525bc3-6784-42a7-9b5b-6c1fe167d454" (UID: "68525bc3-6784-42a7-9b5b-6c1fe167d454"). InnerVolumeSpecName "kube-api-access-blmz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:57:05.290827 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:05.290783 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68525bc3-6784-42a7-9b5b-6c1fe167d454-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:57:05.290827 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:05.290821 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blmz5\" (UniqueName: \"kubernetes.io/projected/68525bc3-6784-42a7-9b5b-6c1fe167d454-kube-api-access-blmz5\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:57:05.904982 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:05.904950 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b5f75e9-7k9qt" Nov 03 20:57:05.905172 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:05.904987 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b5f75e9-7k9qt" event={"ID":"68525bc3-6784-42a7-9b5b-6c1fe167d454","Type":"ContainerDied","Data":"e539650d65f17fd8d9ff7519bc724510160d7b5505130251af5f6cfd0bf18af8"} Nov 03 20:57:05.905172 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:05.905042 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e539650d65f17fd8d9ff7519bc724510160d7b5505130251af5f6cfd0bf18af8" Nov 03 20:57:13.625414 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:13.625372 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b32989ae-4hzzn"] Nov 03 20:57:13.626166 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:13.626148 4305 cpu_manager.go:414] "containerMap: removing container" podUID="68525bc3-6784-42a7-9b5b-6c1fe167d454" containerName="revoke-keys" Nov 03 20:57:13.626218 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:13.626169 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68525bc3-6784-42a7-9b5b-6c1fe167d454" containerName="revoke-keys" Nov 03 20:57:13.626263 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:13.626254 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="68525bc3-6784-42a7-9b5b-6c1fe167d454" containerName="revoke-keys" Nov 03 20:57:13.630089 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:13.630069 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b32989ae-4hzzn" Nov 03 20:57:13.682232 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:13.682188 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:57:13.687681 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:13.687646 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b32989ae-4hzzn"] Nov 03 20:57:13.693311 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:13.693277 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbcc1750-05a6-41a4-b0ea-0390b5ba7d77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b32989ae-4hzzn\" (UID: \"bbcc1750-05a6-41a4-b0ea-0390b5ba7d77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b32989ae-4hzzn" Nov 03 20:57:13.693521 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:13.693418 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv92b\" (UniqueName: \"kubernetes.io/projected/bbcc1750-05a6-41a4-b0ea-0390b5ba7d77-kube-api-access-gv92b\") pod \"maas-api-revoke-sub-b32989ae-4hzzn\" (UID: \"bbcc1750-05a6-41a4-b0ea-0390b5ba7d77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b32989ae-4hzzn" Nov 03 20:57:13.693659 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:13.693640 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbcc1750-05a6-41a4-b0ea-0390b5ba7d77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b32989ae-4hzzn\" (UID: \"bbcc1750-05a6-41a4-b0ea-0390b5ba7d77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b32989ae-4hzzn" Nov 03 20:57:13.731373 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:13.731339 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:57:13.731373 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:13.731365 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77vtd\"" Nov 03 20:57:13.731373 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:13.731339 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:57:13.795370 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:13.795330 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbcc1750-05a6-41a4-b0ea-0390b5ba7d77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b32989ae-4hzzn\" (UID: \"bbcc1750-05a6-41a4-b0ea-0390b5ba7d77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b32989ae-4hzzn" Nov 03 20:57:13.795613 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:13.795406 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv92b\" (UniqueName: \"kubernetes.io/projected/bbcc1750-05a6-41a4-b0ea-0390b5ba7d77-kube-api-access-gv92b\") pod \"maas-api-revoke-sub-b32989ae-4hzzn\" (UID: \"bbcc1750-05a6-41a4-b0ea-0390b5ba7d77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b32989ae-4hzzn" Nov 03 20:57:13.795613 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:13.795540 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbcc1750-05a6-41a4-b0ea-0390b5ba7d77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b32989ae-4hzzn\" (UID: \"bbcc1750-05a6-41a4-b0ea-0390b5ba7d77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b32989ae-4hzzn" Nov 03 20:57:13.796248 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:13.796208 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbcc1750-05a6-41a4-b0ea-0390b5ba7d77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b32989ae-4hzzn\" (UID: \"bbcc1750-05a6-41a4-b0ea-0390b5ba7d77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b32989ae-4hzzn" Nov 03 20:57:13.798105 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:13.798080 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbcc1750-05a6-41a4-b0ea-0390b5ba7d77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b32989ae-4hzzn\" (UID: \"bbcc1750-05a6-41a4-b0ea-0390b5ba7d77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b32989ae-4hzzn" Nov 03 20:57:13.858404 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:13.858374 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv92b\" (UniqueName: \"kubernetes.io/projected/bbcc1750-05a6-41a4-b0ea-0390b5ba7d77-kube-api-access-gv92b\") pod \"maas-api-revoke-sub-b32989ae-4hzzn\" (UID: \"bbcc1750-05a6-41a4-b0ea-0390b5ba7d77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b32989ae-4hzzn" Nov 03 20:57:13.945032 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:13.944989 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b32989ae-4hzzn" Nov 03 20:57:14.329839 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:57:14.329808 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbcc1750_05a6_41a4_b0ea_0390b5ba7d77.slice/crio-a7eafc83c96b8dc53595e4cd63a8b6eeec09742525e246f6786e1380a5357a37 WatchSource:0}: Error finding container a7eafc83c96b8dc53595e4cd63a8b6eeec09742525e246f6786e1380a5357a37: Status 404 returned error can't find the container with id a7eafc83c96b8dc53595e4cd63a8b6eeec09742525e246f6786e1380a5357a37 Nov 03 20:57:14.330483 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:14.330460 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b32989ae-4hzzn"] Nov 03 20:57:14.973275 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:14.973178 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b32989ae-4hzzn" event={"ID":"bbcc1750-05a6-41a4-b0ea-0390b5ba7d77","Type":"ContainerStarted","Data":"ffca4bda878a750313991f2567c01b70d82e86bbccc678b68fe0b30d264c2b37"} Nov 03 20:57:14.973275 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:14.973240 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b32989ae-4hzzn" event={"ID":"bbcc1750-05a6-41a4-b0ea-0390b5ba7d77","Type":"ContainerStarted","Data":"a7eafc83c96b8dc53595e4cd63a8b6eeec09742525e246f6786e1380a5357a37"} Nov 03 20:57:15.107909 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:15.107849 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b32989ae-4hzzn" podStartSLOduration=2.107823522 podStartE2EDuration="2.107823522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:57:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:57:15.105839905 +0000 UTC m=+1070.331159695" watchObservedRunningTime="2026-11-03 20:57:15.107823522 +0000 UTC m=+1070.333143313" Nov 03 20:57:15.982120 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:15.982086 4305 generic.go:356] "Generic (PLEG): container finished" podID="bbcc1750-05a6-41a4-b0ea-0390b5ba7d77" containerID="ffca4bda878a750313991f2567c01b70d82e86bbccc678b68fe0b30d264c2b37" exitCode=0 Nov 03 20:57:15.982546 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:15.982163 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b32989ae-4hzzn" event={"ID":"bbcc1750-05a6-41a4-b0ea-0390b5ba7d77","Type":"ContainerDied","Data":"ffca4bda878a750313991f2567c01b70d82e86bbccc678b68fe0b30d264c2b37"} Nov 03 20:57:16.322875 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:57:16.322786 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 20:57:17.152564 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:17.152529 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b32989ae-4hzzn" Nov 03 20:57:17.236227 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:17.236188 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbcc1750-05a6-41a4-b0ea-0390b5ba7d77-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbcc1750-05a6-41a4-b0ea-0390b5ba7d77-maas-api-xks-ca\") pod \"bbcc1750-05a6-41a4-b0ea-0390b5ba7d77\" (UID: \"bbcc1750-05a6-41a4-b0ea-0390b5ba7d77\") " Nov 03 20:57:17.236429 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:17.236289 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbcc1750-05a6-41a4-b0ea-0390b5ba7d77-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbcc1750-05a6-41a4-b0ea-0390b5ba7d77-maas-api-openshift-ca\") pod \"bbcc1750-05a6-41a4-b0ea-0390b5ba7d77\" (UID: \"bbcc1750-05a6-41a4-b0ea-0390b5ba7d77\") " Nov 03 20:57:17.236429 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:17.236330 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbcc1750-05a6-41a4-b0ea-0390b5ba7d77-kube-api-access-gv92b\" (UniqueName: \"kubernetes.io/projected/bbcc1750-05a6-41a4-b0ea-0390b5ba7d77-kube-api-access-gv92b\") pod \"bbcc1750-05a6-41a4-b0ea-0390b5ba7d77\" (UID: \"bbcc1750-05a6-41a4-b0ea-0390b5ba7d77\") " Nov 03 20:57:17.236759 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:17.236734 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbcc1750-05a6-41a4-b0ea-0390b5ba7d77-maas-api-openshift-ca" pod "bbcc1750-05a6-41a4-b0ea-0390b5ba7d77" (UID: "bbcc1750-05a6-41a4-b0ea-0390b5ba7d77"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:57:17.238708 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:17.238641 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbcc1750-05a6-41a4-b0ea-0390b5ba7d77-maas-api-xks-ca" pod "bbcc1750-05a6-41a4-b0ea-0390b5ba7d77" (UID: "bbcc1750-05a6-41a4-b0ea-0390b5ba7d77"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:57:17.238708 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:17.238664 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbcc1750-05a6-41a4-b0ea-0390b5ba7d77-kube-api-access-gv92b" pod "bbcc1750-05a6-41a4-b0ea-0390b5ba7d77" (UID: "bbcc1750-05a6-41a4-b0ea-0390b5ba7d77"). InnerVolumeSpecName "kube-api-access-gv92b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:57:17.338234 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:17.338182 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbcc1750-05a6-41a4-b0ea-0390b5ba7d77-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:57:17.338234 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:17.338238 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbcc1750-05a6-41a4-b0ea-0390b5ba7d77-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:57:17.338494 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:17.338254 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv92b\" (UniqueName: \"kubernetes.io/projected/bbcc1750-05a6-41a4-b0ea-0390b5ba7d77-kube-api-access-gv92b\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:57:17.813041 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:17.813007 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9029abd-84sxf"] Nov 03 20:57:17.813613 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:17.813598 4305 cpu_manager.go:414] "containerMap: removing container" podUID="bbcc1750-05a6-41a4-b0ea-0390b5ba7d77" containerName="revoke-keys" Nov 03 20:57:17.813613 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:17.813613 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbcc1750-05a6-41a4-b0ea-0390b5ba7d77" containerName="revoke-keys" Nov 03 20:57:17.813750 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:17.813723 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbcc1750-05a6-41a4-b0ea-0390b5ba7d77" containerName="revoke-keys" Nov 03 20:57:17.817374 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:17.817354 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9029abd-84sxf" Nov 03 20:57:17.875398 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:17.875366 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9029abd-84sxf"] Nov 03 20:57:17.946076 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:17.946043 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11edcf66-04da-4a99-a3d8-67635dd2ff0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9029abd-84sxf\" (UID: \"11edcf66-04da-4a99-a3d8-67635dd2ff0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9029abd-84sxf" Nov 03 20:57:17.946257 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:17.946215 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwjpp\" (UniqueName: \"kubernetes.io/projected/11edcf66-04da-4a99-a3d8-67635dd2ff0b-kube-api-access-gwjpp\") pod \"maas-api-revoke-sub-d9029abd-84sxf\" (UID: \"11edcf66-04da-4a99-a3d8-67635dd2ff0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9029abd-84sxf" Nov 03 20:57:17.946296 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:17.946266 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11edcf66-04da-4a99-a3d8-67635dd2ff0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9029abd-84sxf\" (UID: \"11edcf66-04da-4a99-a3d8-67635dd2ff0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9029abd-84sxf" Nov 03 20:57:17.960942 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:17.960908 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt"] Nov 03 20:57:17.965188 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:17.965165 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" Nov 03 20:57:17.998514 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:17.998411 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b32989ae-4hzzn" event={"ID":"bbcc1750-05a6-41a4-b0ea-0390b5ba7d77","Type":"ContainerDied","Data":"a7eafc83c96b8dc53595e4cd63a8b6eeec09742525e246f6786e1380a5357a37"} Nov 03 20:57:17.998514 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:17.998479 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b32989ae-4hzzn" Nov 03 20:57:17.998514 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:17.998501 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7eafc83c96b8dc53595e4cd63a8b6eeec09742525e246f6786e1380a5357a37" Nov 03 20:57:18.022245 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:18.022216 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-d01cef\"/\"kube-root-ca.crt\"" Nov 03 20:57:18.022506 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:18.022216 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-d01cef\"/\"openshift-service-ca.crt\"" Nov 03 20:57:18.022639 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:18.022413 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-d01cef\"/\"distinct-w1-d01cef-kserve-self-signed-certs\"" Nov 03 20:57:18.022731 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:18.022598 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-d01cef\"/\"default-dockercfg-2qc48\"" Nov 03 20:57:18.027383 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:18.027348 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt"] Nov 03 20:57:18.047538 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:18.047495 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scs4x\" (UniqueName: \"kubernetes.io/projected/6e7f30a9-0862-48a5-9ab7-e692e75bb294-kube-api-access-scs4x\") pod \"distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt\" (UID: \"6e7f30a9-0862-48a5-9ab7-e692e75bb294\") " pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" Nov 03 20:57:18.047733 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:18.047569 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11edcf66-04da-4a99-a3d8-67635dd2ff0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9029abd-84sxf\" (UID: \"11edcf66-04da-4a99-a3d8-67635dd2ff0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9029abd-84sxf" Nov 03 20:57:18.047733 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:18.047692 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e7f30a9-0862-48a5-9ab7-e692e75bb294-tls-certs\") pod \"distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt\" (UID: \"6e7f30a9-0862-48a5-9ab7-e692e75bb294\") " pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" Nov 03 20:57:18.047824 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:18.047789 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e7f30a9-0862-48a5-9ab7-e692e75bb294-model-cache\") pod \"distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt\" (UID: \"6e7f30a9-0862-48a5-9ab7-e692e75bb294\") " pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" Nov 03 20:57:18.047824 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:18.047814 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e7f30a9-0862-48a5-9ab7-e692e75bb294-home\") pod \"distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt\" (UID: \"6e7f30a9-0862-48a5-9ab7-e692e75bb294\") " pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" Nov 03 20:57:18.048004 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:18.047985 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwjpp\" (UniqueName: \"kubernetes.io/projected/11edcf66-04da-4a99-a3d8-67635dd2ff0b-kube-api-access-gwjpp\") pod \"maas-api-revoke-sub-d9029abd-84sxf\" (UID: \"11edcf66-04da-4a99-a3d8-67635dd2ff0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9029abd-84sxf" Nov 03 20:57:18.048065 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:18.048021 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e7f30a9-0862-48a5-9ab7-e692e75bb294-dshm\") pod \"distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt\" (UID: \"6e7f30a9-0862-48a5-9ab7-e692e75bb294\") " pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" Nov 03 20:57:18.048122 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:18.048094 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11edcf66-04da-4a99-a3d8-67635dd2ff0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9029abd-84sxf\" (UID: \"11edcf66-04da-4a99-a3d8-67635dd2ff0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9029abd-84sxf" Nov 03 20:57:18.048178 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:18.048127 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e7f30a9-0862-48a5-9ab7-e692e75bb294-tmp-dir\") pod \"distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt\" (UID: \"6e7f30a9-0862-48a5-9ab7-e692e75bb294\") " pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" Nov 03 20:57:18.048526 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:18.048498 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11edcf66-04da-4a99-a3d8-67635dd2ff0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9029abd-84sxf\" (UID: \"11edcf66-04da-4a99-a3d8-67635dd2ff0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9029abd-84sxf" Nov 03 20:57:18.050872 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:18.050847 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11edcf66-04da-4a99-a3d8-67635dd2ff0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9029abd-84sxf\" (UID: \"11edcf66-04da-4a99-a3d8-67635dd2ff0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9029abd-84sxf" Nov 03 20:57:18.110230 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:18.110134 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwjpp\" (UniqueName: \"kubernetes.io/projected/11edcf66-04da-4a99-a3d8-67635dd2ff0b-kube-api-access-gwjpp\") pod \"maas-api-revoke-sub-d9029abd-84sxf\" (UID: \"11edcf66-04da-4a99-a3d8-67635dd2ff0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9029abd-84sxf" Nov 03 20:57:18.131698 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:18.131659 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9029abd-84sxf" Nov 03 20:57:18.149644 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:18.149609 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e7f30a9-0862-48a5-9ab7-e692e75bb294-home\") pod \"distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt\" (UID: \"6e7f30a9-0862-48a5-9ab7-e692e75bb294\") " pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" Nov 03 20:57:18.149761 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:18.149746 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e7f30a9-0862-48a5-9ab7-e692e75bb294-dshm\") pod \"distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt\" (UID: \"6e7f30a9-0862-48a5-9ab7-e692e75bb294\") " pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" Nov 03 20:57:18.149831 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:18.149817 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e7f30a9-0862-48a5-9ab7-e692e75bb294-tmp-dir\") pod \"distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt\" (UID: \"6e7f30a9-0862-48a5-9ab7-e692e75bb294\") " pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" Nov 03 20:57:18.149918 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:18.149895 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scs4x\" (UniqueName: \"kubernetes.io/projected/6e7f30a9-0862-48a5-9ab7-e692e75bb294-kube-api-access-scs4x\") pod \"distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt\" (UID: \"6e7f30a9-0862-48a5-9ab7-e692e75bb294\") " pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" Nov 03 20:57:18.150062 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:18.149979 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e7f30a9-0862-48a5-9ab7-e692e75bb294-tls-certs\") pod \"distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt\" (UID: \"6e7f30a9-0862-48a5-9ab7-e692e75bb294\") " pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" Nov 03 20:57:18.150062 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:18.149987 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e7f30a9-0862-48a5-9ab7-e692e75bb294-home\") pod \"distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt\" (UID: \"6e7f30a9-0862-48a5-9ab7-e692e75bb294\") " pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" Nov 03 20:57:18.150062 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:18.150056 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e7f30a9-0862-48a5-9ab7-e692e75bb294-model-cache\") pod \"distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt\" (UID: \"6e7f30a9-0862-48a5-9ab7-e692e75bb294\") " pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" Nov 03 20:57:18.150221 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:18.150202 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e7f30a9-0862-48a5-9ab7-e692e75bb294-tmp-dir\") pod \"distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt\" (UID: \"6e7f30a9-0862-48a5-9ab7-e692e75bb294\") " pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" Nov 03 20:57:18.150393 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:18.150373 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e7f30a9-0862-48a5-9ab7-e692e75bb294-model-cache\") pod \"distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt\" (UID: \"6e7f30a9-0862-48a5-9ab7-e692e75bb294\") " pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" Nov 03 20:57:18.152803 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:18.152770 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e7f30a9-0862-48a5-9ab7-e692e75bb294-dshm\") pod \"distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt\" (UID: \"6e7f30a9-0862-48a5-9ab7-e692e75bb294\") " pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" Nov 03 20:57:18.153131 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:18.152950 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e7f30a9-0862-48a5-9ab7-e692e75bb294-tls-certs\") pod \"distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt\" (UID: \"6e7f30a9-0862-48a5-9ab7-e692e75bb294\") " pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" Nov 03 20:57:18.213473 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:18.213378 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scs4x\" (UniqueName: \"kubernetes.io/projected/6e7f30a9-0862-48a5-9ab7-e692e75bb294-kube-api-access-scs4x\") pod \"distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt\" (UID: \"6e7f30a9-0862-48a5-9ab7-e692e75bb294\") " pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" Nov 03 20:57:18.278635 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:18.278598 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" Nov 03 20:57:18.528372 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:18.528345 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9029abd-84sxf"] Nov 03 20:57:18.528992 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:57:18.528966 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11edcf66_04da_4a99_a3d8_67635dd2ff0b.slice/crio-418b1139963b893b94318fa1f38a92e4e6f56e2254a91192f33af6d2428c42e9 WatchSource:0}: Error finding container 418b1139963b893b94318fa1f38a92e4e6f56e2254a91192f33af6d2428c42e9: Status 404 returned error can't find the container with id 418b1139963b893b94318fa1f38a92e4e6f56e2254a91192f33af6d2428c42e9 Nov 03 20:57:18.686725 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:18.686693 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt"] Nov 03 20:57:18.686884 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:57:18.686862 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e7f30a9_0862_48a5_9ab7_e692e75bb294.slice/crio-266df45a42582bd33c725e3cdfb2dc05a86f9b4bd8c1f2df563429af679ee2a5 WatchSource:0}: Error finding container 266df45a42582bd33c725e3cdfb2dc05a86f9b4bd8c1f2df563429af679ee2a5: Status 404 returned error can't find the container with id 266df45a42582bd33c725e3cdfb2dc05a86f9b4bd8c1f2df563429af679ee2a5 Nov 03 20:57:19.006283 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:19.006176 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9029abd-84sxf" event={"ID":"11edcf66-04da-4a99-a3d8-67635dd2ff0b","Type":"ContainerStarted","Data":"4e35ffdeab57a1483b3e9c61515f0b5ab2791da4589c005380386852f0e6b067"} Nov 03 20:57:19.006283 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:19.006237 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9029abd-84sxf" event={"ID":"11edcf66-04da-4a99-a3d8-67635dd2ff0b","Type":"ContainerStarted","Data":"418b1139963b893b94318fa1f38a92e4e6f56e2254a91192f33af6d2428c42e9"} Nov 03 20:57:19.007898 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:19.007876 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" event={"ID":"6e7f30a9-0862-48a5-9ab7-e692e75bb294","Type":"ContainerStarted","Data":"0628e6efd4338d498f7287cd26233a950f96353d5c6ec3bc3ce3c285a3039816"} Nov 03 20:57:19.008058 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:19.007907 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" event={"ID":"6e7f30a9-0862-48a5-9ab7-e692e75bb294","Type":"ContainerStarted","Data":"266df45a42582bd33c725e3cdfb2dc05a86f9b4bd8c1f2df563429af679ee2a5"} Nov 03 20:57:19.125725 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:19.125662 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9029abd-84sxf" podStartSLOduration=2.125646059 podStartE2EDuration="2.125646059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:57:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:57:19.124723341 +0000 UTC m=+1074.350043135" watchObservedRunningTime="2026-11-03 20:57:19.125646059 +0000 UTC m=+1074.350965849" Nov 03 20:57:19.244206 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:19.244143 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" podStartSLOduration=2.244121396 podStartE2EDuration="2.244121396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:57:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:57:19.240645512 +0000 UTC m=+1074.465965320" watchObservedRunningTime="2026-11-03 20:57:19.244121396 +0000 UTC m=+1074.469441187" Nov 03 20:57:19.278937 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:19.278845 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" Nov 03 20:57:19.791478 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:19.791421 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh"] Nov 03 20:57:19.795686 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:19.795658 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh" Nov 03 20:57:19.848059 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:19.848024 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-499187-openshift-default-dockercfg-jsmm9\"" Nov 03 20:57:19.856891 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:19.856857 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh"] Nov 03 20:57:19.880719 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:19.880685 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1-istio-token\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh\" (UID: \"17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh" Nov 03 20:57:19.880719 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:19.880728 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1-workload-certs\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh\" (UID: \"17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh" Nov 03 20:57:19.880982 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:19.880822 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1-istio-podinfo\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh\" (UID: \"17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh" Nov 03 20:57:19.880982 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:19.880862 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlqfk\" (UniqueName: \"kubernetes.io/projected/17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1-kube-api-access-mlqfk\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh\" (UID: \"17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh" Nov 03 20:57:19.881090 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:19.881067 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1-istiod-ca-cert\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh\" (UID: \"17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh" Nov 03 20:57:19.881168 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:19.881154 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1-credential-socket\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh\" (UID: \"17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh" Nov 03 20:57:19.881237 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:19.881225 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1-workload-socket\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh\" (UID: \"17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh" Nov 03 20:57:19.881300 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:19.881262 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1-istio-envoy\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh\" (UID: \"17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh" Nov 03 20:57:19.881300 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:19.881288 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1-istio-data\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh\" (UID: \"17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh" Nov 03 20:57:19.982583 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:19.982552 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1-istiod-ca-cert\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh\" (UID: \"17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh" Nov 03 20:57:19.982796 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:19.982665 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1-credential-socket\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh\" (UID: \"17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh" Nov 03 20:57:19.982796 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:19.982748 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1-workload-socket\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh\" (UID: \"17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh" Nov 03 20:57:19.982796 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:19.982782 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1-istio-envoy\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh\" (UID: \"17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh" Nov 03 20:57:19.982995 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:19.982809 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1-istio-data\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh\" (UID: \"17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh" Nov 03 20:57:19.982995 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:19.982885 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1-istio-token\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh\" (UID: \"17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh" Nov 03 20:57:19.982995 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:19.982912 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1-workload-certs\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh\" (UID: \"17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh" Nov 03 20:57:19.982995 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:19.982951 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1-istio-podinfo\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh\" (UID: \"17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh" Nov 03 20:57:19.983188 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:19.982999 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlqfk\" (UniqueName: \"kubernetes.io/projected/17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1-kube-api-access-mlqfk\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh\" (UID: \"17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh" Nov 03 20:57:19.983188 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:19.983172 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1-workload-socket\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh\" (UID: \"17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh" Nov 03 20:57:19.983395 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:19.983351 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1-istiod-ca-cert\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh\" (UID: \"17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh" Nov 03 20:57:19.983563 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:19.983515 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1-istio-data\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh\" (UID: \"17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh" Nov 03 20:57:19.983753 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:19.983733 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1-workload-certs\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh\" (UID: \"17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh" Nov 03 20:57:19.983861 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:19.983800 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1-credential-socket\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh\" (UID: \"17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh" Nov 03 20:57:19.985870 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:19.985842 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1-istio-envoy\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh\" (UID: \"17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh" Nov 03 20:57:19.986039 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:19.986018 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1-istio-podinfo\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh\" (UID: \"17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh" Nov 03 20:57:20.016721 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:20.016686 4305 generic.go:356] "Generic (PLEG): container finished" podID="11edcf66-04da-4a99-a3d8-67635dd2ff0b" containerID="4e35ffdeab57a1483b3e9c61515f0b5ab2791da4589c005380386852f0e6b067" exitCode=0 Nov 03 20:57:20.016903 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:20.016774 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9029abd-84sxf" event={"ID":"11edcf66-04da-4a99-a3d8-67635dd2ff0b","Type":"ContainerDied","Data":"4e35ffdeab57a1483b3e9c61515f0b5ab2791da4589c005380386852f0e6b067"} Nov 03 20:57:20.044743 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:20.044664 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1-istio-token\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh\" (UID: \"17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh" Nov 03 20:57:20.044896 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:20.044746 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlqfk\" (UniqueName: \"kubernetes.io/projected/17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1-kube-api-access-mlqfk\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh\" (UID: \"17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh" Nov 03 20:57:20.109592 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:20.109552 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh" Nov 03 20:57:20.611273 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:57:20.611238 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17bbc6a8_9ea0_4b75_a88c_b15525fb0dc1.slice/crio-4b7db1844b8f7e2c6850a6de34987e8a0be37f36e4ab613887063b938b0024de WatchSource:0}: Error finding container 4b7db1844b8f7e2c6850a6de34987e8a0be37f36e4ab613887063b938b0024de: Status 404 returned error can't find the container with id 4b7db1844b8f7e2c6850a6de34987e8a0be37f36e4ab613887063b938b0024de Nov 03 20:57:20.611697 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:20.611305 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh"] Nov 03 20:57:20.614271 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:20.614236 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:57:20.614349 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:20.614308 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:57:20.614349 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:20.614339 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:57:21.026702 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:21.026665 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh" event={"ID":"17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1","Type":"ContainerStarted","Data":"ea7e5f654bccbde6382c9f018f3f2ec5f2bc78514d2af0f94c66559e0d9aae46"} Nov 03 20:57:21.026906 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:21.026721 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh" event={"ID":"17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1","Type":"ContainerStarted","Data":"4b7db1844b8f7e2c6850a6de34987e8a0be37f36e4ab613887063b938b0024de"} Nov 03 20:57:21.123979 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:21.123357 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh" Nov 03 20:57:21.124221 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:21.124184 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 03 20:57:21.124311 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:21.124242 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh" podUID="17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 03 20:57:21.148349 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:21.148295 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh" podStartSLOduration=2.148278161 podStartE2EDuration="2.148278161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:57:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:57:21.146232258 +0000 UTC m=+1076.371552050" watchObservedRunningTime="2026-11-03 20:57:21.148278161 +0000 UTC m=+1076.373597951" Nov 03 20:57:21.211780 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:21.211756 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9029abd-84sxf" Nov 03 20:57:21.305208 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:21.305108 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11edcf66-04da-4a99-a3d8-67635dd2ff0b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11edcf66-04da-4a99-a3d8-67635dd2ff0b-maas-api-openshift-ca\") pod \"11edcf66-04da-4a99-a3d8-67635dd2ff0b\" (UID: \"11edcf66-04da-4a99-a3d8-67635dd2ff0b\") " Nov 03 20:57:21.305208 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:21.305166 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11edcf66-04da-4a99-a3d8-67635dd2ff0b-kube-api-access-gwjpp\" (UniqueName: \"kubernetes.io/projected/11edcf66-04da-4a99-a3d8-67635dd2ff0b-kube-api-access-gwjpp\") pod \"11edcf66-04da-4a99-a3d8-67635dd2ff0b\" (UID: \"11edcf66-04da-4a99-a3d8-67635dd2ff0b\") " Nov 03 20:57:21.305499 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:21.305222 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11edcf66-04da-4a99-a3d8-67635dd2ff0b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11edcf66-04da-4a99-a3d8-67635dd2ff0b-maas-api-xks-ca\") pod \"11edcf66-04da-4a99-a3d8-67635dd2ff0b\" (UID: \"11edcf66-04da-4a99-a3d8-67635dd2ff0b\") " Nov 03 20:57:21.305575 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:21.305500 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11edcf66-04da-4a99-a3d8-67635dd2ff0b-maas-api-openshift-ca" pod "11edcf66-04da-4a99-a3d8-67635dd2ff0b" (UID: "11edcf66-04da-4a99-a3d8-67635dd2ff0b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:57:21.305749 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:21.305732 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11edcf66-04da-4a99-a3d8-67635dd2ff0b-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:57:21.307530 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:21.307505 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11edcf66-04da-4a99-a3d8-67635dd2ff0b-kube-api-access-gwjpp" pod "11edcf66-04da-4a99-a3d8-67635dd2ff0b" (UID: "11edcf66-04da-4a99-a3d8-67635dd2ff0b"). InnerVolumeSpecName "kube-api-access-gwjpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:57:21.307636 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:21.307619 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11edcf66-04da-4a99-a3d8-67635dd2ff0b-maas-api-xks-ca" pod "11edcf66-04da-4a99-a3d8-67635dd2ff0b" (UID: "11edcf66-04da-4a99-a3d8-67635dd2ff0b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:57:21.407679 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:21.407637 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwjpp\" (UniqueName: \"kubernetes.io/projected/11edcf66-04da-4a99-a3d8-67635dd2ff0b-kube-api-access-gwjpp\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:57:21.407679 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:21.407685 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11edcf66-04da-4a99-a3d8-67635dd2ff0b-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:57:22.043747 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:22.043718 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9029abd-84sxf" Nov 03 20:57:22.044156 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:22.043754 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9029abd-84sxf" event={"ID":"11edcf66-04da-4a99-a3d8-67635dd2ff0b","Type":"ContainerDied","Data":"418b1139963b893b94318fa1f38a92e4e6f56e2254a91192f33af6d2428c42e9"} Nov 03 20:57:22.044156 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:22.043821 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="418b1139963b893b94318fa1f38a92e4e6f56e2254a91192f33af6d2428c42e9" Nov 03 20:57:22.110874 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:22.110838 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 03 20:57:22.111035 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:22.110896 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh" podUID="17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 03 20:57:23.110832 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:23.110798 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 03 20:57:23.111242 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:23.110859 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh" podUID="17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 03 20:57:24.110384 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:24.110351 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 03 20:57:24.110606 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:24.110399 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh" podUID="17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 03 20:57:25.091912 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:25.091877 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 20:57:25.110842 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:25.110804 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 03 20:57:25.111021 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:25.110854 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh" podUID="17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 03 20:57:26.116464 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:26.114277 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 03 20:57:26.116464 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:26.114332 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh" podUID="17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 03 20:57:27.111457 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:27.110557 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 03 20:57:27.111457 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:27.110639 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh" podUID="17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 03 20:57:28.115915 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:28.115887 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh" Nov 03 20:57:28.116371 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:28.116351 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh" Nov 03 20:57:28.117385 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:28.117362 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh" Nov 03 20:57:28.278984 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:28.278946 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" Nov 03 20:57:28.300530 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:28.300496 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" Nov 03 20:57:29.160458 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:29.157212 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" Nov 03 20:57:35.078473 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:35.078414 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr"] Nov 03 20:57:35.079277 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:35.079254 4305 cpu_manager.go:414] "containerMap: removing container" podUID="11edcf66-04da-4a99-a3d8-67635dd2ff0b" containerName="revoke-keys" Nov 03 20:57:35.079375 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:35.079279 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11edcf66-04da-4a99-a3d8-67635dd2ff0b" containerName="revoke-keys" Nov 03 20:57:35.080067 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:35.080034 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="11edcf66-04da-4a99-a3d8-67635dd2ff0b" containerName="revoke-keys" Nov 03 20:57:35.088210 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:35.088179 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" Nov 03 20:57:35.141006 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:35.140974 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr"] Nov 03 20:57:35.185625 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:35.185595 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-499187\"/\"distinct-w0-499187-kserve-self-signed-certs\"" Nov 03 20:57:35.188660 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:35.188632 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ee1f028-8503-4034-9954-eb06fcf402e7-dshm\") pod \"distinct-w0-499187-kserve-8644578469-4zqnr\" (UID: \"8ee1f028-8503-4034-9954-eb06fcf402e7\") " pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" Nov 03 20:57:35.188781 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:35.188685 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ee1f028-8503-4034-9954-eb06fcf402e7-home\") pod \"distinct-w0-499187-kserve-8644578469-4zqnr\" (UID: \"8ee1f028-8503-4034-9954-eb06fcf402e7\") " pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" Nov 03 20:57:35.188781 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:35.188761 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ee1f028-8503-4034-9954-eb06fcf402e7-tmp-dir\") pod \"distinct-w0-499187-kserve-8644578469-4zqnr\" (UID: \"8ee1f028-8503-4034-9954-eb06fcf402e7\") " pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" Nov 03 20:57:35.188887 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:35.188867 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ee1f028-8503-4034-9954-eb06fcf402e7-tls-certs\") pod \"distinct-w0-499187-kserve-8644578469-4zqnr\" (UID: \"8ee1f028-8503-4034-9954-eb06fcf402e7\") " pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" Nov 03 20:57:35.188997 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:35.188985 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5hl5\" (UniqueName: \"kubernetes.io/projected/8ee1f028-8503-4034-9954-eb06fcf402e7-kube-api-access-g5hl5\") pod \"distinct-w0-499187-kserve-8644578469-4zqnr\" (UID: \"8ee1f028-8503-4034-9954-eb06fcf402e7\") " pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" Nov 03 20:57:35.189069 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:35.189057 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ee1f028-8503-4034-9954-eb06fcf402e7-model-cache\") pod \"distinct-w0-499187-kserve-8644578469-4zqnr\" (UID: \"8ee1f028-8503-4034-9954-eb06fcf402e7\") " pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" Nov 03 20:57:35.290036 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:35.289999 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ee1f028-8503-4034-9954-eb06fcf402e7-model-cache\") pod \"distinct-w0-499187-kserve-8644578469-4zqnr\" (UID: \"8ee1f028-8503-4034-9954-eb06fcf402e7\") " pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" Nov 03 20:57:35.290036 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:35.290049 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ee1f028-8503-4034-9954-eb06fcf402e7-dshm\") pod \"distinct-w0-499187-kserve-8644578469-4zqnr\" (UID: \"8ee1f028-8503-4034-9954-eb06fcf402e7\") " pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" Nov 03 20:57:35.290295 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:35.290101 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ee1f028-8503-4034-9954-eb06fcf402e7-home\") pod \"distinct-w0-499187-kserve-8644578469-4zqnr\" (UID: \"8ee1f028-8503-4034-9954-eb06fcf402e7\") " pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" Nov 03 20:57:35.290295 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:35.290128 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ee1f028-8503-4034-9954-eb06fcf402e7-tmp-dir\") pod \"distinct-w0-499187-kserve-8644578469-4zqnr\" (UID: \"8ee1f028-8503-4034-9954-eb06fcf402e7\") " pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" Nov 03 20:57:35.290295 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:35.290249 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ee1f028-8503-4034-9954-eb06fcf402e7-tls-certs\") pod \"distinct-w0-499187-kserve-8644578469-4zqnr\" (UID: \"8ee1f028-8503-4034-9954-eb06fcf402e7\") " pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" Nov 03 20:57:35.290474 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:35.290397 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5hl5\" (UniqueName: \"kubernetes.io/projected/8ee1f028-8503-4034-9954-eb06fcf402e7-kube-api-access-g5hl5\") pod \"distinct-w0-499187-kserve-8644578469-4zqnr\" (UID: \"8ee1f028-8503-4034-9954-eb06fcf402e7\") " pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" Nov 03 20:57:35.290821 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:35.290792 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ee1f028-8503-4034-9954-eb06fcf402e7-home\") pod \"distinct-w0-499187-kserve-8644578469-4zqnr\" (UID: \"8ee1f028-8503-4034-9954-eb06fcf402e7\") " pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" Nov 03 20:57:35.290968 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:35.290915 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ee1f028-8503-4034-9954-eb06fcf402e7-tmp-dir\") pod \"distinct-w0-499187-kserve-8644578469-4zqnr\" (UID: \"8ee1f028-8503-4034-9954-eb06fcf402e7\") " pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" Nov 03 20:57:35.290968 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:35.290941 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ee1f028-8503-4034-9954-eb06fcf402e7-model-cache\") pod \"distinct-w0-499187-kserve-8644578469-4zqnr\" (UID: \"8ee1f028-8503-4034-9954-eb06fcf402e7\") " pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" Nov 03 20:57:35.292954 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:35.292933 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ee1f028-8503-4034-9954-eb06fcf402e7-dshm\") pod \"distinct-w0-499187-kserve-8644578469-4zqnr\" (UID: \"8ee1f028-8503-4034-9954-eb06fcf402e7\") " pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" Nov 03 20:57:35.293084 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:35.293066 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ee1f028-8503-4034-9954-eb06fcf402e7-tls-certs\") pod \"distinct-w0-499187-kserve-8644578469-4zqnr\" (UID: \"8ee1f028-8503-4034-9954-eb06fcf402e7\") " pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" Nov 03 20:57:35.351112 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:35.351043 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5hl5\" (UniqueName: \"kubernetes.io/projected/8ee1f028-8503-4034-9954-eb06fcf402e7-kube-api-access-g5hl5\") pod \"distinct-w0-499187-kserve-8644578469-4zqnr\" (UID: \"8ee1f028-8503-4034-9954-eb06fcf402e7\") " pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" Nov 03 20:57:35.405363 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:35.405313 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" Nov 03 20:57:35.804945 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:57:35.804910 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8ee1f028_8503_4034_9954_eb06fcf402e7.slice/crio-3623c894e94a7aeff03abc649813f12557f459ad647bcc40bc73c1d8223a0501 WatchSource:0}: Error finding container 3623c894e94a7aeff03abc649813f12557f459ad647bcc40bc73c1d8223a0501: Status 404 returned error can't find the container with id 3623c894e94a7aeff03abc649813f12557f459ad647bcc40bc73c1d8223a0501 Nov 03 20:57:35.805365 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:35.805341 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr"] Nov 03 20:57:36.198664 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:36.198627 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" event={"ID":"8ee1f028-8503-4034-9954-eb06fcf402e7","Type":"ContainerStarted","Data":"af86062f6aee09ba31fc2e3ec73039f924e12d291f1051c73faa10d4218e16c6"} Nov 03 20:57:36.199103 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:36.198674 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" event={"ID":"8ee1f028-8503-4034-9954-eb06fcf402e7","Type":"ContainerStarted","Data":"3623c894e94a7aeff03abc649813f12557f459ad647bcc40bc73c1d8223a0501"} Nov 03 20:57:36.319407 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:36.319352 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" podStartSLOduration=1.319334988 podStartE2EDuration="1.319334988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:57:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:57:36.316476012 +0000 UTC m=+1091.541795806" watchObservedRunningTime="2026-11-03 20:57:36.319334988 +0000 UTC m=+1091.544654778" Nov 03 20:57:36.405733 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:36.405697 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" Nov 03 20:57:45.405581 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:45.405544 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" Nov 03 20:57:45.418582 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:45.418556 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" Nov 03 20:57:46.277386 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:57:46.277352 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" Nov 03 20:57:46.387602 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:57:46.387563 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 20:58:00.874311 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:00.874275 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 20:58:05.151784 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:05.151753 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x"] Nov 03 20:58:05.157198 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:05.157176 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x" Nov 03 20:58:05.262117 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:05.261903 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x"] Nov 03 20:58:05.269238 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:05.265192 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1920ffc0-36e7-4d99-9410-444ad24afd9c-credential-socket\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x\" (UID: \"1920ffc0-36e7-4d99-9410-444ad24afd9c\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x" Nov 03 20:58:05.269238 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:05.265242 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj55h\" (UniqueName: \"kubernetes.io/projected/1920ffc0-36e7-4d99-9410-444ad24afd9c-kube-api-access-gj55h\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x\" (UID: \"1920ffc0-36e7-4d99-9410-444ad24afd9c\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x" Nov 03 20:58:05.269238 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:05.265316 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1920ffc0-36e7-4d99-9410-444ad24afd9c-istio-data\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x\" (UID: \"1920ffc0-36e7-4d99-9410-444ad24afd9c\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x" Nov 03 20:58:05.269238 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:05.265342 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1920ffc0-36e7-4d99-9410-444ad24afd9c-istio-podinfo\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x\" (UID: \"1920ffc0-36e7-4d99-9410-444ad24afd9c\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x" Nov 03 20:58:05.269238 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:05.265382 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1920ffc0-36e7-4d99-9410-444ad24afd9c-workload-certs\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x\" (UID: \"1920ffc0-36e7-4d99-9410-444ad24afd9c\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x" Nov 03 20:58:05.269238 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:05.265468 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1920ffc0-36e7-4d99-9410-444ad24afd9c-istio-envoy\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x\" (UID: \"1920ffc0-36e7-4d99-9410-444ad24afd9c\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x" Nov 03 20:58:05.269238 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:05.265539 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1920ffc0-36e7-4d99-9410-444ad24afd9c-istiod-ca-cert\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x\" (UID: \"1920ffc0-36e7-4d99-9410-444ad24afd9c\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x" Nov 03 20:58:05.269238 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:05.265836 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1920ffc0-36e7-4d99-9410-444ad24afd9c-workload-socket\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x\" (UID: \"1920ffc0-36e7-4d99-9410-444ad24afd9c\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x" Nov 03 20:58:05.269238 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:05.266068 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1920ffc0-36e7-4d99-9410-444ad24afd9c-istio-token\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x\" (UID: \"1920ffc0-36e7-4d99-9410-444ad24afd9c\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x" Nov 03 20:58:05.366905 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:05.366866 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1920ffc0-36e7-4d99-9410-444ad24afd9c-istiod-ca-cert\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x\" (UID: \"1920ffc0-36e7-4d99-9410-444ad24afd9c\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x" Nov 03 20:58:05.367081 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:05.366920 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1920ffc0-36e7-4d99-9410-444ad24afd9c-workload-socket\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x\" (UID: \"1920ffc0-36e7-4d99-9410-444ad24afd9c\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x" Nov 03 20:58:05.367081 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:05.366954 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1920ffc0-36e7-4d99-9410-444ad24afd9c-istio-token\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x\" (UID: \"1920ffc0-36e7-4d99-9410-444ad24afd9c\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x" Nov 03 20:58:05.367081 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:05.367022 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1920ffc0-36e7-4d99-9410-444ad24afd9c-credential-socket\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x\" (UID: \"1920ffc0-36e7-4d99-9410-444ad24afd9c\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x" Nov 03 20:58:05.367081 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:05.367042 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj55h\" (UniqueName: \"kubernetes.io/projected/1920ffc0-36e7-4d99-9410-444ad24afd9c-kube-api-access-gj55h\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x\" (UID: \"1920ffc0-36e7-4d99-9410-444ad24afd9c\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x" Nov 03 20:58:05.367272 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:05.367099 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1920ffc0-36e7-4d99-9410-444ad24afd9c-istio-data\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x\" (UID: \"1920ffc0-36e7-4d99-9410-444ad24afd9c\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x" Nov 03 20:58:05.367272 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:05.367123 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1920ffc0-36e7-4d99-9410-444ad24afd9c-istio-podinfo\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x\" (UID: \"1920ffc0-36e7-4d99-9410-444ad24afd9c\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x" Nov 03 20:58:05.367272 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:05.367159 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1920ffc0-36e7-4d99-9410-444ad24afd9c-workload-certs\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x\" (UID: \"1920ffc0-36e7-4d99-9410-444ad24afd9c\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x" Nov 03 20:58:05.367272 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:05.367220 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1920ffc0-36e7-4d99-9410-444ad24afd9c-istio-envoy\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x\" (UID: \"1920ffc0-36e7-4d99-9410-444ad24afd9c\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x" Nov 03 20:58:05.367527 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:05.367370 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1920ffc0-36e7-4d99-9410-444ad24afd9c-workload-socket\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x\" (UID: \"1920ffc0-36e7-4d99-9410-444ad24afd9c\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x" Nov 03 20:58:05.367527 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:05.367406 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1920ffc0-36e7-4d99-9410-444ad24afd9c-credential-socket\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x\" (UID: \"1920ffc0-36e7-4d99-9410-444ad24afd9c\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x" Nov 03 20:58:05.367527 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:05.367517 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1920ffc0-36e7-4d99-9410-444ad24afd9c-istio-data\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x\" (UID: \"1920ffc0-36e7-4d99-9410-444ad24afd9c\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x" Nov 03 20:58:05.367690 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:05.367665 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1920ffc0-36e7-4d99-9410-444ad24afd9c-workload-certs\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x\" (UID: \"1920ffc0-36e7-4d99-9410-444ad24afd9c\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x" Nov 03 20:58:05.367690 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:05.367681 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1920ffc0-36e7-4d99-9410-444ad24afd9c-istiod-ca-cert\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x\" (UID: \"1920ffc0-36e7-4d99-9410-444ad24afd9c\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x" Nov 03 20:58:05.369624 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:05.369607 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1920ffc0-36e7-4d99-9410-444ad24afd9c-istio-envoy\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x\" (UID: \"1920ffc0-36e7-4d99-9410-444ad24afd9c\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x" Nov 03 20:58:05.369851 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:05.369833 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1920ffc0-36e7-4d99-9410-444ad24afd9c-istio-podinfo\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x\" (UID: \"1920ffc0-36e7-4d99-9410-444ad24afd9c\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x" Nov 03 20:58:05.428103 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:05.428026 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1920ffc0-36e7-4d99-9410-444ad24afd9c-istio-token\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x\" (UID: \"1920ffc0-36e7-4d99-9410-444ad24afd9c\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x" Nov 03 20:58:05.428103 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:05.428071 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj55h\" (UniqueName: \"kubernetes.io/projected/1920ffc0-36e7-4d99-9410-444ad24afd9c-kube-api-access-gj55h\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x\" (UID: \"1920ffc0-36e7-4d99-9410-444ad24afd9c\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x" Nov 03 20:58:05.471914 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:05.471884 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x" Nov 03 20:58:05.806318 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:05.806234 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-54764"] Nov 03 20:58:05.811960 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:05.811941 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-54764" Nov 03 20:58:05.873238 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:05.873204 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-54764"] Nov 03 20:58:05.873838 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:05.873814 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c9106f1-54e4-4c54-bba9-938e47117a27-credential-socket\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-54764\" (UID: \"5c9106f1-54e4-4c54-bba9-938e47117a27\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-54764" Nov 03 20:58:05.873906 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:05.873860 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c9106f1-54e4-4c54-bba9-938e47117a27-istio-data\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-54764\" (UID: \"5c9106f1-54e4-4c54-bba9-938e47117a27\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-54764" Nov 03 20:58:05.874021 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:05.874001 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjqsd\" (UniqueName: \"kubernetes.io/projected/5c9106f1-54e4-4c54-bba9-938e47117a27-kube-api-access-tjqsd\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-54764\" (UID: \"5c9106f1-54e4-4c54-bba9-938e47117a27\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-54764" Nov 03 20:58:05.874066 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:05.874050 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c9106f1-54e4-4c54-bba9-938e47117a27-istio-token\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-54764\" (UID: \"5c9106f1-54e4-4c54-bba9-938e47117a27\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-54764" Nov 03 20:58:05.874160 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:05.874140 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c9106f1-54e4-4c54-bba9-938e47117a27-istiod-ca-cert\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-54764\" (UID: \"5c9106f1-54e4-4c54-bba9-938e47117a27\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-54764" Nov 03 20:58:05.874217 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:05.874206 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c9106f1-54e4-4c54-bba9-938e47117a27-workload-socket\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-54764\" (UID: \"5c9106f1-54e4-4c54-bba9-938e47117a27\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-54764" Nov 03 20:58:05.874312 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:05.874301 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c9106f1-54e4-4c54-bba9-938e47117a27-workload-certs\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-54764\" (UID: \"5c9106f1-54e4-4c54-bba9-938e47117a27\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-54764" Nov 03 20:58:05.874349 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:05.874319 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c9106f1-54e4-4c54-bba9-938e47117a27-istio-envoy\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-54764\" (UID: \"5c9106f1-54e4-4c54-bba9-938e47117a27\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-54764" Nov 03 20:58:05.874385 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:05.874352 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c9106f1-54e4-4c54-bba9-938e47117a27-istio-podinfo\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-54764\" (UID: \"5c9106f1-54e4-4c54-bba9-938e47117a27\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-54764" Nov 03 20:58:05.906626 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:05.906595 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x"] Nov 03 20:58:05.907383 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:58:05.907358 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1920ffc0_36e7_4d99_9410_444ad24afd9c.slice/crio-e27c9be914b46c36f33b2a1a0878b4957735860465de2ef23f15725c354f363d WatchSource:0}: Error finding container e27c9be914b46c36f33b2a1a0878b4957735860465de2ef23f15725c354f363d: Status 404 returned error can't find the container with id e27c9be914b46c36f33b2a1a0878b4957735860465de2ef23f15725c354f363d Nov 03 20:58:05.910794 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:05.910758 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:58:05.910905 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:05.910822 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:58:05.910905 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:05.910852 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:58:05.975872 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:05.975844 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c9106f1-54e4-4c54-bba9-938e47117a27-istiod-ca-cert\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-54764\" (UID: \"5c9106f1-54e4-4c54-bba9-938e47117a27\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-54764" Nov 03 20:58:05.976019 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:05.975909 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c9106f1-54e4-4c54-bba9-938e47117a27-workload-socket\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-54764\" (UID: \"5c9106f1-54e4-4c54-bba9-938e47117a27\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-54764" Nov 03 20:58:05.976019 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:05.975961 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c9106f1-54e4-4c54-bba9-938e47117a27-workload-certs\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-54764\" (UID: \"5c9106f1-54e4-4c54-bba9-938e47117a27\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-54764" Nov 03 20:58:05.976019 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:05.975977 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c9106f1-54e4-4c54-bba9-938e47117a27-istio-envoy\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-54764\" (UID: \"5c9106f1-54e4-4c54-bba9-938e47117a27\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-54764" Nov 03 20:58:05.976163 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:05.976022 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c9106f1-54e4-4c54-bba9-938e47117a27-istio-podinfo\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-54764\" (UID: \"5c9106f1-54e4-4c54-bba9-938e47117a27\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-54764" Nov 03 20:58:05.976163 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:05.976089 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c9106f1-54e4-4c54-bba9-938e47117a27-credential-socket\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-54764\" (UID: \"5c9106f1-54e4-4c54-bba9-938e47117a27\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-54764" Nov 03 20:58:05.976163 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:05.976125 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c9106f1-54e4-4c54-bba9-938e47117a27-istio-data\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-54764\" (UID: \"5c9106f1-54e4-4c54-bba9-938e47117a27\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-54764" Nov 03 20:58:05.976300 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:05.976202 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjqsd\" (UniqueName: \"kubernetes.io/projected/5c9106f1-54e4-4c54-bba9-938e47117a27-kube-api-access-tjqsd\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-54764\" (UID: \"5c9106f1-54e4-4c54-bba9-938e47117a27\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-54764" Nov 03 20:58:05.976556 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:05.976543 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c9106f1-54e4-4c54-bba9-938e47117a27-istio-token\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-54764\" (UID: \"5c9106f1-54e4-4c54-bba9-938e47117a27\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-54764" Nov 03 20:58:05.976610 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:05.976572 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c9106f1-54e4-4c54-bba9-938e47117a27-workload-socket\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-54764\" (UID: \"5c9106f1-54e4-4c54-bba9-938e47117a27\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-54764" Nov 03 20:58:05.976774 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:05.976739 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c9106f1-54e4-4c54-bba9-938e47117a27-credential-socket\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-54764\" (UID: \"5c9106f1-54e4-4c54-bba9-938e47117a27\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-54764" Nov 03 20:58:05.976844 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:05.976806 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c9106f1-54e4-4c54-bba9-938e47117a27-istio-data\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-54764\" (UID: \"5c9106f1-54e4-4c54-bba9-938e47117a27\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-54764" Nov 03 20:58:05.976898 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:05.976875 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c9106f1-54e4-4c54-bba9-938e47117a27-istiod-ca-cert\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-54764\" (UID: \"5c9106f1-54e4-4c54-bba9-938e47117a27\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-54764" Nov 03 20:58:05.976955 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:05.976915 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c9106f1-54e4-4c54-bba9-938e47117a27-workload-certs\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-54764\" (UID: \"5c9106f1-54e4-4c54-bba9-938e47117a27\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-54764" Nov 03 20:58:05.978795 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:05.978773 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c9106f1-54e4-4c54-bba9-938e47117a27-istio-envoy\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-54764\" (UID: \"5c9106f1-54e4-4c54-bba9-938e47117a27\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-54764" Nov 03 20:58:05.979045 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:05.979025 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c9106f1-54e4-4c54-bba9-938e47117a27-istio-podinfo\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-54764\" (UID: \"5c9106f1-54e4-4c54-bba9-938e47117a27\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-54764" Nov 03 20:58:06.037318 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:06.037281 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjqsd\" (UniqueName: \"kubernetes.io/projected/5c9106f1-54e4-4c54-bba9-938e47117a27-kube-api-access-tjqsd\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-54764\" (UID: \"5c9106f1-54e4-4c54-bba9-938e47117a27\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-54764" Nov 03 20:58:06.042006 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:06.041978 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c9106f1-54e4-4c54-bba9-938e47117a27-istio-token\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-54764\" (UID: \"5c9106f1-54e4-4c54-bba9-938e47117a27\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-54764" Nov 03 20:58:06.127964 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:06.127876 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-54764" Nov 03 20:58:06.415519 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:06.415482 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x" event={"ID":"1920ffc0-36e7-4d99-9410-444ad24afd9c","Type":"ContainerStarted","Data":"dfbf78fe82e05e2a90b38ad21dac6ac7e2793aa394447992c7651e802a95478d"} Nov 03 20:58:06.416052 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:06.415543 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x" event={"ID":"1920ffc0-36e7-4d99-9410-444ad24afd9c","Type":"ContainerStarted","Data":"e27c9be914b46c36f33b2a1a0878b4957735860465de2ef23f15725c354f363d"} Nov 03 20:58:06.473457 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:06.472654 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x" Nov 03 20:58:06.479070 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:06.479035 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 20:58:06.479353 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:06.479324 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x" podUID="1920ffc0-36e7-4d99-9410-444ad24afd9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 20:58:06.544466 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:06.544400 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x" podStartSLOduration=1.5443847609999999 podStartE2EDuration="1.544384761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:58:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:58:06.540928886 +0000 UTC m=+1121.766248720" watchObservedRunningTime="2026-11-03 20:58:06.544384761 +0000 UTC m=+1121.769704551" Nov 03 20:58:06.579751 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:06.579722 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-54764"] Nov 03 20:58:06.582856 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:58:06.582810 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c9106f1_54e4_4c54_bba9_938e47117a27.slice/crio-304393eb01df074cafe626c70bdcd44b35066904f8515ee2d7d7430e2bdaf0f2 WatchSource:0}: Error finding container 304393eb01df074cafe626c70bdcd44b35066904f8515ee2d7d7430e2bdaf0f2: Status 404 returned error can't find the container with id 304393eb01df074cafe626c70bdcd44b35066904f8515ee2d7d7430e2bdaf0f2 Nov 03 20:58:06.585760 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:06.585722 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:58:06.585879 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:06.585807 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:58:06.585879 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:06.585854 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:58:07.426743 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:07.426709 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-54764" event={"ID":"5c9106f1-54e4-4c54-bba9-938e47117a27","Type":"ContainerStarted","Data":"bd80f226c3b8f9ec219a2624fb20a6a867fb13a000a19135a95b550af3da72ca"} Nov 03 20:58:07.427186 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:07.426755 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-54764" event={"ID":"5c9106f1-54e4-4c54-bba9-938e47117a27","Type":"ContainerStarted","Data":"304393eb01df074cafe626c70bdcd44b35066904f8515ee2d7d7430e2bdaf0f2"} Nov 03 20:58:07.475460 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:07.472872 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 20:58:07.475460 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:07.472932 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x" podUID="1920ffc0-36e7-4d99-9410-444ad24afd9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 20:58:07.600460 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:07.600247 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-54764" podStartSLOduration=2.6002239 podStartE2EDuration="2.6002239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:58:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:58:07.574930195 +0000 UTC m=+1122.800250062" watchObservedRunningTime="2026-11-03 20:58:07.6002239 +0000 UTC m=+1122.825543694" Nov 03 20:58:08.128401 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:08.128362 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-54764" Nov 03 20:58:08.129952 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:08.129928 4305 patch_prober.go:29] interesting pod/e2e-worker-w2-862138-openshift-default-bc56c55fc-54764 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 20:58:08.130109 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:08.129984 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-54764" podUID="5c9106f1-54e4-4c54-bba9-938e47117a27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 20:58:08.476458 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:08.475914 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 20:58:08.476458 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:08.475976 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x" podUID="1920ffc0-36e7-4d99-9410-444ad24afd9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 20:58:09.129090 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:09.129045 4305 patch_prober.go:29] interesting pod/e2e-worker-w2-862138-openshift-default-bc56c55fc-54764 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 20:58:09.129296 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:09.129116 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-54764" podUID="5c9106f1-54e4-4c54-bba9-938e47117a27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 20:58:09.473459 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:09.472673 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 20:58:09.473459 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:09.472735 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x" podUID="1920ffc0-36e7-4d99-9410-444ad24afd9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 20:58:10.129337 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:10.129297 4305 patch_prober.go:29] interesting pod/e2e-worker-w2-862138-openshift-default-bc56c55fc-54764 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 20:58:10.129858 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:10.129355 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-54764" podUID="5c9106f1-54e4-4c54-bba9-938e47117a27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 20:58:10.477214 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:10.477175 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 20:58:10.477700 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:10.477667 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x" podUID="1920ffc0-36e7-4d99-9410-444ad24afd9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 20:58:11.128608 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:11.128571 4305 patch_prober.go:29] interesting pod/e2e-worker-w2-862138-openshift-default-bc56c55fc-54764 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 20:58:11.128799 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:11.128639 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-54764" podUID="5c9106f1-54e4-4c54-bba9-938e47117a27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 20:58:11.478013 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:11.477979 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 20:58:11.501777 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:11.501734 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x" podUID="1920ffc0-36e7-4d99-9410-444ad24afd9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 20:58:12.128405 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:12.128367 4305 patch_prober.go:29] interesting pod/e2e-worker-w2-862138-openshift-default-bc56c55fc-54764 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 20:58:12.128636 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:12.128425 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-54764" podUID="5c9106f1-54e4-4c54-bba9-938e47117a27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 20:58:12.482466 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:12.479924 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Nov 03 20:58:12.482466 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:12.480003 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x" podUID="1920ffc0-36e7-4d99-9410-444ad24afd9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Nov 03 20:58:13.128642 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:13.128598 4305 patch_prober.go:29] interesting pod/e2e-worker-w2-862138-openshift-default-bc56c55fc-54764 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 20:58:13.128867 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:13.128673 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-54764" podUID="5c9106f1-54e4-4c54-bba9-938e47117a27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 20:58:13.478816 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:13.478784 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x" Nov 03 20:58:13.479243 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:13.479223 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x" Nov 03 20:58:13.479982 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:13.479964 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x" Nov 03 20:58:14.133548 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:14.133520 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-54764" Nov 03 20:58:14.133991 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:14.133974 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-54764" Nov 03 20:58:14.134817 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:14.134797 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-54764" Nov 03 20:58:16.322500 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:58:16.322464 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 20:58:37.853353 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:58:37.853317 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 20:58:46.330083 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:58:46.330044 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 20:59:00.990602 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:00.990571 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26"] Nov 03 20:59:00.994938 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:00.994913 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" Nov 03 20:59:01.096994 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:01.096957 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-d01cef\"/\"distinct-2-w1-d01cef-kserve-self-signed-certs\"" Nov 03 20:59:01.101663 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:01.101631 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26"] Nov 03 20:59:01.111727 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:01.111701 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5d3dcaa-a51a-40c6-8049-871b300552e1-tls-certs\") pod \"distinct-2-w1-d01cef-kserve-547bffcc96-scw26\" (UID: \"d5d3dcaa-a51a-40c6-8049-871b300552e1\") " pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" Nov 03 20:59:01.111902 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:01.111871 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5d3dcaa-a51a-40c6-8049-871b300552e1-home\") pod \"distinct-2-w1-d01cef-kserve-547bffcc96-scw26\" (UID: \"d5d3dcaa-a51a-40c6-8049-871b300552e1\") " pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" Nov 03 20:59:01.111963 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:01.111950 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5d3dcaa-a51a-40c6-8049-871b300552e1-tmp-dir\") pod \"distinct-2-w1-d01cef-kserve-547bffcc96-scw26\" (UID: \"d5d3dcaa-a51a-40c6-8049-871b300552e1\") " pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" Nov 03 20:59:01.112016 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:01.111998 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5d3dcaa-a51a-40c6-8049-871b300552e1-dshm\") pod \"distinct-2-w1-d01cef-kserve-547bffcc96-scw26\" (UID: \"d5d3dcaa-a51a-40c6-8049-871b300552e1\") " pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" Nov 03 20:59:01.112121 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:01.112106 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwgpv\" (UniqueName: \"kubernetes.io/projected/d5d3dcaa-a51a-40c6-8049-871b300552e1-kube-api-access-bwgpv\") pod \"distinct-2-w1-d01cef-kserve-547bffcc96-scw26\" (UID: \"d5d3dcaa-a51a-40c6-8049-871b300552e1\") " pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" Nov 03 20:59:01.112215 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:01.112202 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5d3dcaa-a51a-40c6-8049-871b300552e1-model-cache\") pod \"distinct-2-w1-d01cef-kserve-547bffcc96-scw26\" (UID: \"d5d3dcaa-a51a-40c6-8049-871b300552e1\") " pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" Nov 03 20:59:01.213021 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:01.212986 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5d3dcaa-a51a-40c6-8049-871b300552e1-home\") pod \"distinct-2-w1-d01cef-kserve-547bffcc96-scw26\" (UID: \"d5d3dcaa-a51a-40c6-8049-871b300552e1\") " pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" Nov 03 20:59:01.213239 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:01.213043 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5d3dcaa-a51a-40c6-8049-871b300552e1-tmp-dir\") pod \"distinct-2-w1-d01cef-kserve-547bffcc96-scw26\" (UID: \"d5d3dcaa-a51a-40c6-8049-871b300552e1\") " pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" Nov 03 20:59:01.213239 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:01.213062 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5d3dcaa-a51a-40c6-8049-871b300552e1-dshm\") pod \"distinct-2-w1-d01cef-kserve-547bffcc96-scw26\" (UID: \"d5d3dcaa-a51a-40c6-8049-871b300552e1\") " pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" Nov 03 20:59:01.213239 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:01.213091 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwgpv\" (UniqueName: \"kubernetes.io/projected/d5d3dcaa-a51a-40c6-8049-871b300552e1-kube-api-access-bwgpv\") pod \"distinct-2-w1-d01cef-kserve-547bffcc96-scw26\" (UID: \"d5d3dcaa-a51a-40c6-8049-871b300552e1\") " pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" Nov 03 20:59:01.213239 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:01.213171 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5d3dcaa-a51a-40c6-8049-871b300552e1-model-cache\") pod \"distinct-2-w1-d01cef-kserve-547bffcc96-scw26\" (UID: \"d5d3dcaa-a51a-40c6-8049-871b300552e1\") " pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" Nov 03 20:59:01.213239 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:01.213198 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5d3dcaa-a51a-40c6-8049-871b300552e1-tls-certs\") pod \"distinct-2-w1-d01cef-kserve-547bffcc96-scw26\" (UID: \"d5d3dcaa-a51a-40c6-8049-871b300552e1\") " pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" Nov 03 20:59:01.213559 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:01.213522 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5d3dcaa-a51a-40c6-8049-871b300552e1-home\") pod \"distinct-2-w1-d01cef-kserve-547bffcc96-scw26\" (UID: \"d5d3dcaa-a51a-40c6-8049-871b300552e1\") " pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" Nov 03 20:59:01.213559 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:01.213538 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5d3dcaa-a51a-40c6-8049-871b300552e1-tmp-dir\") pod \"distinct-2-w1-d01cef-kserve-547bffcc96-scw26\" (UID: \"d5d3dcaa-a51a-40c6-8049-871b300552e1\") " pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" Nov 03 20:59:01.213659 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:01.213639 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5d3dcaa-a51a-40c6-8049-871b300552e1-model-cache\") pod \"distinct-2-w1-d01cef-kserve-547bffcc96-scw26\" (UID: \"d5d3dcaa-a51a-40c6-8049-871b300552e1\") " pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" Nov 03 20:59:01.215633 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:01.215601 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5d3dcaa-a51a-40c6-8049-871b300552e1-dshm\") pod \"distinct-2-w1-d01cef-kserve-547bffcc96-scw26\" (UID: \"d5d3dcaa-a51a-40c6-8049-871b300552e1\") " pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" Nov 03 20:59:01.215798 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:01.215783 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5d3dcaa-a51a-40c6-8049-871b300552e1-tls-certs\") pod \"distinct-2-w1-d01cef-kserve-547bffcc96-scw26\" (UID: \"d5d3dcaa-a51a-40c6-8049-871b300552e1\") " pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" Nov 03 20:59:01.277559 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:01.277486 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwgpv\" (UniqueName: \"kubernetes.io/projected/d5d3dcaa-a51a-40c6-8049-871b300552e1-kube-api-access-bwgpv\") pod \"distinct-2-w1-d01cef-kserve-547bffcc96-scw26\" (UID: \"d5d3dcaa-a51a-40c6-8049-871b300552e1\") " pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" Nov 03 20:59:01.307278 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:01.307246 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" Nov 03 20:59:01.696291 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:01.696263 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26"] Nov 03 20:59:01.696477 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:59:01.696455 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd5d3dcaa_a51a_40c6_8049_871b300552e1.slice/crio-21606bf960577ecfcf3e5eecdc322ed2e1f89849c44a2964dac1931fa4fdb52f WatchSource:0}: Error finding container 21606bf960577ecfcf3e5eecdc322ed2e1f89849c44a2964dac1931fa4fdb52f: Status 404 returned error can't find the container with id 21606bf960577ecfcf3e5eecdc322ed2e1f89849c44a2964dac1931fa4fdb52f Nov 03 20:59:01.882922 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:01.882877 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" event={"ID":"d5d3dcaa-a51a-40c6-8049-871b300552e1","Type":"ContainerStarted","Data":"f267d2c6e0195a332ca66497d52a5a5879c135891e181fe0a9c588980f421940"} Nov 03 20:59:01.883106 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:01.882934 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" event={"ID":"d5d3dcaa-a51a-40c6-8049-871b300552e1","Type":"ContainerStarted","Data":"21606bf960577ecfcf3e5eecdc322ed2e1f89849c44a2964dac1931fa4fdb52f"} Nov 03 20:59:02.000665 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:02.000560 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" podStartSLOduration=2.000544857 podStartE2EDuration="2.000544857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:59:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:59:01.999506223 +0000 UTC m=+1177.224826035" watchObservedRunningTime="2026-11-03 20:59:02.000544857 +0000 UTC m=+1177.225864648" Nov 03 20:59:02.308199 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:02.308113 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" Nov 03 20:59:05.268497 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:05.268465 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s"] Nov 03 20:59:05.272963 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:05.272946 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s" Nov 03 20:59:05.368704 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:05.368653 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7f2f4c8-63bb-4a07-8860-91a420ebc1ae-workload-socket\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s\" (UID: \"c7f2f4c8-63bb-4a07-8860-91a420ebc1ae\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s" Nov 03 20:59:05.368915 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:05.368746 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7f2f4c8-63bb-4a07-8860-91a420ebc1ae-istio-podinfo\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s\" (UID: \"c7f2f4c8-63bb-4a07-8860-91a420ebc1ae\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s" Nov 03 20:59:05.368915 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:05.368883 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7f2f4c8-63bb-4a07-8860-91a420ebc1ae-istiod-ca-cert\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s\" (UID: \"c7f2f4c8-63bb-4a07-8860-91a420ebc1ae\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s" Nov 03 20:59:05.369061 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:05.369016 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7f2f4c8-63bb-4a07-8860-91a420ebc1ae-credential-socket\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s\" (UID: \"c7f2f4c8-63bb-4a07-8860-91a420ebc1ae\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s" Nov 03 20:59:05.369268 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:05.369244 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7f2f4c8-63bb-4a07-8860-91a420ebc1ae-istio-envoy\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s\" (UID: \"c7f2f4c8-63bb-4a07-8860-91a420ebc1ae\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s" Nov 03 20:59:05.369398 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:05.369327 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7f2f4c8-63bb-4a07-8860-91a420ebc1ae-istio-token\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s\" (UID: \"c7f2f4c8-63bb-4a07-8860-91a420ebc1ae\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s" Nov 03 20:59:05.369520 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:05.369423 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7jsx\" (UniqueName: \"kubernetes.io/projected/c7f2f4c8-63bb-4a07-8860-91a420ebc1ae-kube-api-access-b7jsx\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s\" (UID: \"c7f2f4c8-63bb-4a07-8860-91a420ebc1ae\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s" Nov 03 20:59:05.369583 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:05.369549 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7f2f4c8-63bb-4a07-8860-91a420ebc1ae-workload-certs\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s\" (UID: \"c7f2f4c8-63bb-4a07-8860-91a420ebc1ae\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s" Nov 03 20:59:05.369666 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:05.369646 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7f2f4c8-63bb-4a07-8860-91a420ebc1ae-istio-data\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s\" (UID: \"c7f2f4c8-63bb-4a07-8860-91a420ebc1ae\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s" Nov 03 20:59:05.375177 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:05.375151 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s"] Nov 03 20:59:05.470991 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:05.470959 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7f2f4c8-63bb-4a07-8860-91a420ebc1ae-istio-data\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s\" (UID: \"c7f2f4c8-63bb-4a07-8860-91a420ebc1ae\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s" Nov 03 20:59:05.471194 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:05.471012 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7f2f4c8-63bb-4a07-8860-91a420ebc1ae-workload-socket\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s\" (UID: \"c7f2f4c8-63bb-4a07-8860-91a420ebc1ae\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s" Nov 03 20:59:05.471194 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:05.471030 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7f2f4c8-63bb-4a07-8860-91a420ebc1ae-istio-podinfo\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s\" (UID: \"c7f2f4c8-63bb-4a07-8860-91a420ebc1ae\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s" Nov 03 20:59:05.471194 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:05.471062 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7f2f4c8-63bb-4a07-8860-91a420ebc1ae-istiod-ca-cert\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s\" (UID: \"c7f2f4c8-63bb-4a07-8860-91a420ebc1ae\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s" Nov 03 20:59:05.471194 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:05.471134 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7f2f4c8-63bb-4a07-8860-91a420ebc1ae-credential-socket\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s\" (UID: \"c7f2f4c8-63bb-4a07-8860-91a420ebc1ae\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s" Nov 03 20:59:05.471194 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:05.471164 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7f2f4c8-63bb-4a07-8860-91a420ebc1ae-istio-envoy\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s\" (UID: \"c7f2f4c8-63bb-4a07-8860-91a420ebc1ae\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s" Nov 03 20:59:05.471480 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:05.471221 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7f2f4c8-63bb-4a07-8860-91a420ebc1ae-istio-token\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s\" (UID: \"c7f2f4c8-63bb-4a07-8860-91a420ebc1ae\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s" Nov 03 20:59:05.471480 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:05.471256 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7jsx\" (UniqueName: \"kubernetes.io/projected/c7f2f4c8-63bb-4a07-8860-91a420ebc1ae-kube-api-access-b7jsx\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s\" (UID: \"c7f2f4c8-63bb-4a07-8860-91a420ebc1ae\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s" Nov 03 20:59:05.471589 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:05.471496 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7f2f4c8-63bb-4a07-8860-91a420ebc1ae-istio-data\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s\" (UID: \"c7f2f4c8-63bb-4a07-8860-91a420ebc1ae\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s" Nov 03 20:59:05.471589 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:05.471509 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7f2f4c8-63bb-4a07-8860-91a420ebc1ae-workload-certs\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s\" (UID: \"c7f2f4c8-63bb-4a07-8860-91a420ebc1ae\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s" Nov 03 20:59:05.471684 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:05.471600 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7f2f4c8-63bb-4a07-8860-91a420ebc1ae-credential-socket\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s\" (UID: \"c7f2f4c8-63bb-4a07-8860-91a420ebc1ae\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s" Nov 03 20:59:05.471909 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:05.471860 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7f2f4c8-63bb-4a07-8860-91a420ebc1ae-workload-certs\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s\" (UID: \"c7f2f4c8-63bb-4a07-8860-91a420ebc1ae\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s" Nov 03 20:59:05.472031 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:05.472008 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7f2f4c8-63bb-4a07-8860-91a420ebc1ae-workload-socket\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s\" (UID: \"c7f2f4c8-63bb-4a07-8860-91a420ebc1ae\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s" Nov 03 20:59:05.472546 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:05.472521 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7f2f4c8-63bb-4a07-8860-91a420ebc1ae-istiod-ca-cert\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s\" (UID: \"c7f2f4c8-63bb-4a07-8860-91a420ebc1ae\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s" Nov 03 20:59:05.473845 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:05.473825 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7f2f4c8-63bb-4a07-8860-91a420ebc1ae-istio-podinfo\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s\" (UID: \"c7f2f4c8-63bb-4a07-8860-91a420ebc1ae\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s" Nov 03 20:59:05.473933 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:05.473885 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7f2f4c8-63bb-4a07-8860-91a420ebc1ae-istio-envoy\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s\" (UID: \"c7f2f4c8-63bb-4a07-8860-91a420ebc1ae\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s" Nov 03 20:59:05.555866 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:05.555781 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7jsx\" (UniqueName: \"kubernetes.io/projected/c7f2f4c8-63bb-4a07-8860-91a420ebc1ae-kube-api-access-b7jsx\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s\" (UID: \"c7f2f4c8-63bb-4a07-8860-91a420ebc1ae\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s" Nov 03 20:59:05.557978 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:05.557947 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7f2f4c8-63bb-4a07-8860-91a420ebc1ae-istio-token\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s\" (UID: \"c7f2f4c8-63bb-4a07-8860-91a420ebc1ae\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s" Nov 03 20:59:05.587725 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:05.587693 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s" Nov 03 20:59:06.020052 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:59:06.020021 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7f2f4c8_63bb_4a07_8860_91a420ebc1ae.slice/crio-6f2ab1445494e6840b5d702a77523203ef049cb8655d5606145aa198b638d09f WatchSource:0}: Error finding container 6f2ab1445494e6840b5d702a77523203ef049cb8655d5606145aa198b638d09f: Status 404 returned error can't find the container with id 6f2ab1445494e6840b5d702a77523203ef049cb8655d5606145aa198b638d09f Nov 03 20:59:06.020384 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:06.020361 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s"] Nov 03 20:59:06.022903 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:06.022869 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:59:06.023008 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:06.022947 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:59:06.023008 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:06.022989 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:59:06.926698 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:06.926662 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s" event={"ID":"c7f2f4c8-63bb-4a07-8860-91a420ebc1ae","Type":"ContainerStarted","Data":"e15906680b84f49e5f809aa4e8071c32342f2e726bfcc00f659bd78fe8bfaa12"} Nov 03 20:59:06.927191 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:06.926754 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s" event={"ID":"c7f2f4c8-63bb-4a07-8860-91a420ebc1ae","Type":"ContainerStarted","Data":"6f2ab1445494e6840b5d702a77523203ef049cb8655d5606145aa198b638d09f"} Nov 03 20:59:07.050579 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:07.050524 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s" podStartSLOduration=2.0505086869999998 podStartE2EDuration="2.050508687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:59:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:59:07.046517025 +0000 UTC m=+1182.271836834" watchObservedRunningTime="2026-11-03 20:59:07.050508687 +0000 UTC m=+1182.275828476" Nov 03 20:59:07.588377 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:07.588340 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s" Nov 03 20:59:07.589909 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:07.589879 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 03 20:59:07.590040 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:07.589945 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s" podUID="c7f2f4c8-63bb-4a07-8860-91a420ebc1ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 03 20:59:08.589083 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:08.589022 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 03 20:59:08.589556 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:08.589100 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s" podUID="c7f2f4c8-63bb-4a07-8860-91a420ebc1ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 03 20:59:09.588126 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:09.588097 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 03 20:59:09.588320 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:09.588153 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s" podUID="c7f2f4c8-63bb-4a07-8860-91a420ebc1ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 03 20:59:10.588715 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:10.588680 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 03 20:59:10.589125 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:10.588744 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s" podUID="c7f2f4c8-63bb-4a07-8860-91a420ebc1ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 03 20:59:11.307951 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:11.307897 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" Nov 03 20:59:11.320602 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:11.320572 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" Nov 03 20:59:11.588177 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:11.588097 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 03 20:59:11.588177 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:11.588155 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s" podUID="c7f2f4c8-63bb-4a07-8860-91a420ebc1ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 03 20:59:12.002525 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:12.002496 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" Nov 03 20:59:12.588721 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:12.588691 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 03 20:59:12.588889 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:12.588740 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s" podUID="c7f2f4c8-63bb-4a07-8860-91a420ebc1ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 03 20:59:13.650137 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:13.650104 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s" Nov 03 20:59:13.650998 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:13.650973 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s" Nov 03 20:59:13.651715 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:13.651700 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s" Nov 03 20:59:16.321568 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:59:16.321528 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 20:59:17.760407 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:17.760369 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 20:59:20.089365 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:20.089325 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9"] Nov 03 20:59:20.092729 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:20.092710 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9" Nov 03 20:59:20.120450 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:20.120407 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f8328f2-ffba-400c-b4e3-c6c62092fefd-istio-podinfo\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9\" (UID: \"9f8328f2-ffba-400c-b4e3-c6c62092fefd\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9" Nov 03 20:59:20.120620 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:20.120604 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f8328f2-ffba-400c-b4e3-c6c62092fefd-workload-socket\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9\" (UID: \"9f8328f2-ffba-400c-b4e3-c6c62092fefd\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9" Nov 03 20:59:20.120664 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:20.120642 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl425\" (UniqueName: \"kubernetes.io/projected/9f8328f2-ffba-400c-b4e3-c6c62092fefd-kube-api-access-nl425\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9\" (UID: \"9f8328f2-ffba-400c-b4e3-c6c62092fefd\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9" Nov 03 20:59:20.120739 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:20.120723 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f8328f2-ffba-400c-b4e3-c6c62092fefd-workload-certs\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9\" (UID: \"9f8328f2-ffba-400c-b4e3-c6c62092fefd\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9" Nov 03 20:59:20.120789 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:20.120775 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f8328f2-ffba-400c-b4e3-c6c62092fefd-istio-envoy\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9\" (UID: \"9f8328f2-ffba-400c-b4e3-c6c62092fefd\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9" Nov 03 20:59:20.120824 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:20.120807 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f8328f2-ffba-400c-b4e3-c6c62092fefd-istio-token\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9\" (UID: \"9f8328f2-ffba-400c-b4e3-c6c62092fefd\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9" Nov 03 20:59:20.120927 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:20.120906 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f8328f2-ffba-400c-b4e3-c6c62092fefd-credential-socket\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9\" (UID: \"9f8328f2-ffba-400c-b4e3-c6c62092fefd\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9" Nov 03 20:59:20.120962 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:20.120944 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f8328f2-ffba-400c-b4e3-c6c62092fefd-istio-data\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9\" (UID: \"9f8328f2-ffba-400c-b4e3-c6c62092fefd\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9" Nov 03 20:59:20.121028 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:20.121016 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f8328f2-ffba-400c-b4e3-c6c62092fefd-istiod-ca-cert\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9\" (UID: \"9f8328f2-ffba-400c-b4e3-c6c62092fefd\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9" Nov 03 20:59:20.200797 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:20.200763 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9"] Nov 03 20:59:20.222144 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:20.222099 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f8328f2-ffba-400c-b4e3-c6c62092fefd-workload-socket\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9\" (UID: \"9f8328f2-ffba-400c-b4e3-c6c62092fefd\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9" Nov 03 20:59:20.222344 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:20.222156 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl425\" (UniqueName: \"kubernetes.io/projected/9f8328f2-ffba-400c-b4e3-c6c62092fefd-kube-api-access-nl425\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9\" (UID: \"9f8328f2-ffba-400c-b4e3-c6c62092fefd\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9" Nov 03 20:59:20.222344 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:20.222204 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f8328f2-ffba-400c-b4e3-c6c62092fefd-workload-certs\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9\" (UID: \"9f8328f2-ffba-400c-b4e3-c6c62092fefd\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9" Nov 03 20:59:20.222632 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:20.222600 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f8328f2-ffba-400c-b4e3-c6c62092fefd-workload-socket\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9\" (UID: \"9f8328f2-ffba-400c-b4e3-c6c62092fefd\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9" Nov 03 20:59:20.222726 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:20.222634 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f8328f2-ffba-400c-b4e3-c6c62092fefd-workload-certs\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9\" (UID: \"9f8328f2-ffba-400c-b4e3-c6c62092fefd\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9" Nov 03 20:59:20.222726 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:20.222636 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f8328f2-ffba-400c-b4e3-c6c62092fefd-istio-envoy\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9\" (UID: \"9f8328f2-ffba-400c-b4e3-c6c62092fefd\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9" Nov 03 20:59:20.222726 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:20.222681 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f8328f2-ffba-400c-b4e3-c6c62092fefd-istio-token\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9\" (UID: \"9f8328f2-ffba-400c-b4e3-c6c62092fefd\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9" Nov 03 20:59:20.222896 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:20.222741 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f8328f2-ffba-400c-b4e3-c6c62092fefd-credential-socket\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9\" (UID: \"9f8328f2-ffba-400c-b4e3-c6c62092fefd\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9" Nov 03 20:59:20.222896 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:20.222774 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f8328f2-ffba-400c-b4e3-c6c62092fefd-istio-data\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9\" (UID: \"9f8328f2-ffba-400c-b4e3-c6c62092fefd\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9" Nov 03 20:59:20.222896 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:20.222840 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f8328f2-ffba-400c-b4e3-c6c62092fefd-istiod-ca-cert\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9\" (UID: \"9f8328f2-ffba-400c-b4e3-c6c62092fefd\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9" Nov 03 20:59:20.222896 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:20.222865 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f8328f2-ffba-400c-b4e3-c6c62092fefd-istio-podinfo\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9\" (UID: \"9f8328f2-ffba-400c-b4e3-c6c62092fefd\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9" Nov 03 20:59:20.223914 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:20.223211 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f8328f2-ffba-400c-b4e3-c6c62092fefd-istio-data\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9\" (UID: \"9f8328f2-ffba-400c-b4e3-c6c62092fefd\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9" Nov 03 20:59:20.223914 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:20.223261 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f8328f2-ffba-400c-b4e3-c6c62092fefd-credential-socket\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9\" (UID: \"9f8328f2-ffba-400c-b4e3-c6c62092fefd\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9" Nov 03 20:59:20.223914 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:20.223855 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f8328f2-ffba-400c-b4e3-c6c62092fefd-istiod-ca-cert\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9\" (UID: \"9f8328f2-ffba-400c-b4e3-c6c62092fefd\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9" Nov 03 20:59:20.227371 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:20.227334 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f8328f2-ffba-400c-b4e3-c6c62092fefd-istio-envoy\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9\" (UID: \"9f8328f2-ffba-400c-b4e3-c6c62092fefd\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9" Nov 03 20:59:20.227848 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:20.227816 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f8328f2-ffba-400c-b4e3-c6c62092fefd-istio-podinfo\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9\" (UID: \"9f8328f2-ffba-400c-b4e3-c6c62092fefd\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9" Nov 03 20:59:20.283973 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:20.283917 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f8328f2-ffba-400c-b4e3-c6c62092fefd-istio-token\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9\" (UID: \"9f8328f2-ffba-400c-b4e3-c6c62092fefd\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9" Nov 03 20:59:20.284119 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:20.284005 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl425\" (UniqueName: \"kubernetes.io/projected/9f8328f2-ffba-400c-b4e3-c6c62092fefd-kube-api-access-nl425\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9\" (UID: \"9f8328f2-ffba-400c-b4e3-c6c62092fefd\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9" Nov 03 20:59:20.406113 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:20.406082 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9" Nov 03 20:59:20.841409 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:59:20.841380 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f8328f2_ffba_400c_b4e3_c6c62092fefd.slice/crio-71e87e5e3771fc21d13a28ce880d4240e6df532d606eccf084ea75040cd67fb9 WatchSource:0}: Error finding container 71e87e5e3771fc21d13a28ce880d4240e6df532d606eccf084ea75040cd67fb9: Status 404 returned error can't find the container with id 71e87e5e3771fc21d13a28ce880d4240e6df532d606eccf084ea75040cd67fb9 Nov 03 20:59:20.841574 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:20.841474 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9"] Nov 03 20:59:20.843806 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:20.843775 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:59:20.843886 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:20.843840 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:59:20.843886 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:20.843868 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:59:21.060336 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:21.060302 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9" event={"ID":"9f8328f2-ffba-400c-b4e3-c6c62092fefd","Type":"ContainerStarted","Data":"bf55d75d372a4c7e620a0ecd4e1abb1258da7cadad6085e78f6fea556ece66bb"} Nov 03 20:59:21.060548 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:21.060358 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9" event={"ID":"9f8328f2-ffba-400c-b4e3-c6c62092fefd","Type":"ContainerStarted","Data":"71e87e5e3771fc21d13a28ce880d4240e6df532d606eccf084ea75040cd67fb9"} Nov 03 20:59:21.184912 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:21.184855 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9" podStartSLOduration=1.184837054 podStartE2EDuration="1.184837054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:59:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:59:21.182277678 +0000 UTC m=+1196.407597492" watchObservedRunningTime="2026-11-03 20:59:21.184837054 +0000 UTC m=+1196.410156849" Nov 03 20:59:21.406367 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:21.406333 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9" Nov 03 20:59:21.408224 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:21.408201 4305 patch_prober.go:29] interesting pod/e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 03 20:59:21.408292 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:21.408250 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9" podUID="9f8328f2-ffba-400c-b4e3-c6c62092fefd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 03 20:59:22.406769 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:22.406731 4305 patch_prober.go:29] interesting pod/e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 03 20:59:22.407266 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:22.406793 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9" podUID="9f8328f2-ffba-400c-b4e3-c6c62092fefd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 03 20:59:23.407359 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:23.407320 4305 patch_prober.go:29] interesting pod/e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 03 20:59:23.407775 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:23.407377 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9" podUID="9f8328f2-ffba-400c-b4e3-c6c62092fefd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 03 20:59:24.407450 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:24.407399 4305 patch_prober.go:29] interesting pod/e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 03 20:59:24.407810 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:24.407472 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9" podUID="9f8328f2-ffba-400c-b4e3-c6c62092fefd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 03 20:59:25.406648 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:25.406595 4305 patch_prober.go:29] interesting pod/e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 03 20:59:25.406839 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:25.406670 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9" podUID="9f8328f2-ffba-400c-b4e3-c6c62092fefd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 03 20:59:26.407180 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:26.407138 4305 patch_prober.go:29] interesting pod/e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 03 20:59:26.407582 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:26.407199 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9" podUID="9f8328f2-ffba-400c-b4e3-c6c62092fefd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 03 20:59:27.407455 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:27.407405 4305 patch_prober.go:29] interesting pod/e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 03 20:59:27.407846 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:27.407489 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9" podUID="9f8328f2-ffba-400c-b4e3-c6c62092fefd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 03 20:59:28.411815 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:28.411786 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9" Nov 03 20:59:28.412336 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:28.412212 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9" Nov 03 20:59:28.412942 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:28.412922 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9" Nov 03 20:59:36.992030 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:36.991697 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz"] Nov 03 20:59:36.998808 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:36.998782 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz" Nov 03 20:59:37.101262 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:37.101221 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz"] Nov 03 20:59:37.169312 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:37.169278 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjg42\" (UniqueName: \"kubernetes.io/projected/f5b60f69-9404-49bd-a072-d04f9b4ccb19-kube-api-access-cjg42\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz\" (UID: \"f5b60f69-9404-49bd-a072-d04f9b4ccb19\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz" Nov 03 20:59:37.169548 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:37.169336 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5b60f69-9404-49bd-a072-d04f9b4ccb19-workload-certs\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz\" (UID: \"f5b60f69-9404-49bd-a072-d04f9b4ccb19\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz" Nov 03 20:59:37.169548 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:37.169418 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5b60f69-9404-49bd-a072-d04f9b4ccb19-istiod-ca-cert\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz\" (UID: \"f5b60f69-9404-49bd-a072-d04f9b4ccb19\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz" Nov 03 20:59:37.169679 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:37.169598 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b60f69-9404-49bd-a072-d04f9b4ccb19-workload-socket\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz\" (UID: \"f5b60f69-9404-49bd-a072-d04f9b4ccb19\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz" Nov 03 20:59:37.169679 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:37.169626 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5b60f69-9404-49bd-a072-d04f9b4ccb19-istio-token\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz\" (UID: \"f5b60f69-9404-49bd-a072-d04f9b4ccb19\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz" Nov 03 20:59:37.169679 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:37.169655 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5b60f69-9404-49bd-a072-d04f9b4ccb19-istio-podinfo\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz\" (UID: \"f5b60f69-9404-49bd-a072-d04f9b4ccb19\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz" Nov 03 20:59:37.169828 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:37.169708 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5b60f69-9404-49bd-a072-d04f9b4ccb19-istio-data\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz\" (UID: \"f5b60f69-9404-49bd-a072-d04f9b4ccb19\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz" Nov 03 20:59:37.169828 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:37.169778 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b60f69-9404-49bd-a072-d04f9b4ccb19-credential-socket\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz\" (UID: \"f5b60f69-9404-49bd-a072-d04f9b4ccb19\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz" Nov 03 20:59:37.169828 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:37.169802 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5b60f69-9404-49bd-a072-d04f9b4ccb19-istio-envoy\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz\" (UID: \"f5b60f69-9404-49bd-a072-d04f9b4ccb19\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz" Nov 03 20:59:37.271030 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:37.270922 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b60f69-9404-49bd-a072-d04f9b4ccb19-workload-socket\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz\" (UID: \"f5b60f69-9404-49bd-a072-d04f9b4ccb19\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz" Nov 03 20:59:37.271030 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:37.270972 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5b60f69-9404-49bd-a072-d04f9b4ccb19-istio-token\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz\" (UID: \"f5b60f69-9404-49bd-a072-d04f9b4ccb19\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz" Nov 03 20:59:37.271030 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:37.271005 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5b60f69-9404-49bd-a072-d04f9b4ccb19-istio-podinfo\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz\" (UID: \"f5b60f69-9404-49bd-a072-d04f9b4ccb19\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz" Nov 03 20:59:37.271316 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:37.271062 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5b60f69-9404-49bd-a072-d04f9b4ccb19-istio-data\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz\" (UID: \"f5b60f69-9404-49bd-a072-d04f9b4ccb19\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz" Nov 03 20:59:37.271316 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:37.271134 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b60f69-9404-49bd-a072-d04f9b4ccb19-credential-socket\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz\" (UID: \"f5b60f69-9404-49bd-a072-d04f9b4ccb19\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz" Nov 03 20:59:37.271316 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:37.271157 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5b60f69-9404-49bd-a072-d04f9b4ccb19-istio-envoy\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz\" (UID: \"f5b60f69-9404-49bd-a072-d04f9b4ccb19\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz" Nov 03 20:59:37.271316 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:37.271212 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjg42\" (UniqueName: \"kubernetes.io/projected/f5b60f69-9404-49bd-a072-d04f9b4ccb19-kube-api-access-cjg42\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz\" (UID: \"f5b60f69-9404-49bd-a072-d04f9b4ccb19\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz" Nov 03 20:59:37.271316 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:37.271258 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5b60f69-9404-49bd-a072-d04f9b4ccb19-workload-certs\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz\" (UID: \"f5b60f69-9404-49bd-a072-d04f9b4ccb19\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz" Nov 03 20:59:37.271606 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:37.271358 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5b60f69-9404-49bd-a072-d04f9b4ccb19-istiod-ca-cert\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz\" (UID: \"f5b60f69-9404-49bd-a072-d04f9b4ccb19\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz" Nov 03 20:59:37.271606 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:37.271386 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b60f69-9404-49bd-a072-d04f9b4ccb19-workload-socket\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz\" (UID: \"f5b60f69-9404-49bd-a072-d04f9b4ccb19\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz" Nov 03 20:59:37.271717 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:37.271660 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b60f69-9404-49bd-a072-d04f9b4ccb19-credential-socket\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz\" (UID: \"f5b60f69-9404-49bd-a072-d04f9b4ccb19\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz" Nov 03 20:59:37.272213 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:37.272191 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5b60f69-9404-49bd-a072-d04f9b4ccb19-istio-data\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz\" (UID: \"f5b60f69-9404-49bd-a072-d04f9b4ccb19\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz" Nov 03 20:59:37.272419 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:37.272400 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5b60f69-9404-49bd-a072-d04f9b4ccb19-workload-certs\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz\" (UID: \"f5b60f69-9404-49bd-a072-d04f9b4ccb19\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz" Nov 03 20:59:37.273068 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:37.273046 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5b60f69-9404-49bd-a072-d04f9b4ccb19-istiod-ca-cert\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz\" (UID: \"f5b60f69-9404-49bd-a072-d04f9b4ccb19\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz" Nov 03 20:59:37.274414 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:37.274391 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5b60f69-9404-49bd-a072-d04f9b4ccb19-istio-envoy\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz\" (UID: \"f5b60f69-9404-49bd-a072-d04f9b4ccb19\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz" Nov 03 20:59:37.274891 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:37.274872 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5b60f69-9404-49bd-a072-d04f9b4ccb19-istio-podinfo\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz\" (UID: \"f5b60f69-9404-49bd-a072-d04f9b4ccb19\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz" Nov 03 20:59:37.352736 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:37.352700 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5b60f69-9404-49bd-a072-d04f9b4ccb19-istio-token\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz\" (UID: \"f5b60f69-9404-49bd-a072-d04f9b4ccb19\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz" Nov 03 20:59:37.352893 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:37.352761 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjg42\" (UniqueName: \"kubernetes.io/projected/f5b60f69-9404-49bd-a072-d04f9b4ccb19-kube-api-access-cjg42\") pod \"e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz\" (UID: \"f5b60f69-9404-49bd-a072-d04f9b4ccb19\") " pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz" Nov 03 20:59:37.372692 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:37.372662 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz" Nov 03 20:59:37.814661 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:37.814618 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz"] Nov 03 20:59:37.845413 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:59:37.845388 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5b60f69_9404_49bd_a072_d04f9b4ccb19.slice/crio-e9f777fbf30b9ec20089e6f5255d90b9dd010f65b5bca1bd0a57c889ccef9d54 WatchSource:0}: Error finding container e9f777fbf30b9ec20089e6f5255d90b9dd010f65b5bca1bd0a57c889ccef9d54: Status 404 returned error can't find the container with id e9f777fbf30b9ec20089e6f5255d90b9dd010f65b5bca1bd0a57c889ccef9d54 Nov 03 20:59:37.847950 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:37.847896 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:59:37.848060 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:37.848003 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:59:37.848060 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:37.848049 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 20:59:38.119532 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:38.119288 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ed16b66-kdsks"] Nov 03 20:59:38.123766 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:38.123740 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed16b66-kdsks" Nov 03 20:59:38.193265 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:38.193221 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:59:38.220235 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:38.220209 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:59:38.220783 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:38.220749 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:59:38.221132 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:38.221117 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77vtd\"" Nov 03 20:59:38.222270 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:38.222249 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57fea02a-046f-4cd2-9373-09d1c47de7d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ed16b66-kdsks\" (UID: \"57fea02a-046f-4cd2-9373-09d1c47de7d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed16b66-kdsks" Nov 03 20:59:38.222659 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:38.222640 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qvwv\" (UniqueName: \"kubernetes.io/projected/57fea02a-046f-4cd2-9373-09d1c47de7d1-kube-api-access-6qvwv\") pod \"maas-api-revoke-sub-7ed16b66-kdsks\" (UID: \"57fea02a-046f-4cd2-9373-09d1c47de7d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed16b66-kdsks" Nov 03 20:59:38.222920 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:38.222904 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57fea02a-046f-4cd2-9373-09d1c47de7d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ed16b66-kdsks\" (UID: \"57fea02a-046f-4cd2-9373-09d1c47de7d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed16b66-kdsks" Nov 03 20:59:38.224832 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:38.224709 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ed16b66-kdsks"] Nov 03 20:59:38.239407 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:38.239374 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz" event={"ID":"f5b60f69-9404-49bd-a072-d04f9b4ccb19","Type":"ContainerStarted","Data":"bb9e0ba1ad5778ed22016235710eb53c0092ea96b8929007d2495fe00e69ae13"} Nov 03 20:59:38.239682 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:38.239469 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz" event={"ID":"f5b60f69-9404-49bd-a072-d04f9b4ccb19","Type":"ContainerStarted","Data":"e9f777fbf30b9ec20089e6f5255d90b9dd010f65b5bca1bd0a57c889ccef9d54"} Nov 03 20:59:38.324755 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:38.324715 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57fea02a-046f-4cd2-9373-09d1c47de7d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ed16b66-kdsks\" (UID: \"57fea02a-046f-4cd2-9373-09d1c47de7d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed16b66-kdsks" Nov 03 20:59:38.324963 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:38.324894 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57fea02a-046f-4cd2-9373-09d1c47de7d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ed16b66-kdsks\" (UID: \"57fea02a-046f-4cd2-9373-09d1c47de7d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed16b66-kdsks" Nov 03 20:59:38.325963 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:38.325060 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qvwv\" (UniqueName: \"kubernetes.io/projected/57fea02a-046f-4cd2-9373-09d1c47de7d1-kube-api-access-6qvwv\") pod \"maas-api-revoke-sub-7ed16b66-kdsks\" (UID: \"57fea02a-046f-4cd2-9373-09d1c47de7d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed16b66-kdsks" Nov 03 20:59:38.326589 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:38.326568 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57fea02a-046f-4cd2-9373-09d1c47de7d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ed16b66-kdsks\" (UID: \"57fea02a-046f-4cd2-9373-09d1c47de7d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed16b66-kdsks" Nov 03 20:59:38.327995 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:38.327977 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57fea02a-046f-4cd2-9373-09d1c47de7d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ed16b66-kdsks\" (UID: \"57fea02a-046f-4cd2-9373-09d1c47de7d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed16b66-kdsks" Nov 03 20:59:38.375515 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:38.373497 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz" Nov 03 20:59:38.375515 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:38.374926 4305 patch_prober.go:29] interesting pod/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 20:59:38.375515 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:38.375000 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz" podUID="f5b60f69-9404-49bd-a072-d04f9b4ccb19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 20:59:38.391487 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:38.390587 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qvwv\" (UniqueName: \"kubernetes.io/projected/57fea02a-046f-4cd2-9373-09d1c47de7d1-kube-api-access-6qvwv\") pod \"maas-api-revoke-sub-7ed16b66-kdsks\" (UID: \"57fea02a-046f-4cd2-9373-09d1c47de7d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed16b66-kdsks" Nov 03 20:59:38.414903 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:38.414838 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz" podStartSLOduration=2.414816853 podStartE2EDuration="2.414816853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:59:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:59:38.407418192 +0000 UTC m=+1213.632738033" watchObservedRunningTime="2026-11-03 20:59:38.414816853 +0000 UTC m=+1213.640136642" Nov 03 20:59:38.439683 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:38.439248 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed16b66-kdsks" Nov 03 20:59:38.879061 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:38.879032 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ed16b66-kdsks"] Nov 03 20:59:38.881753 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:59:38.881718 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57fea02a_046f_4cd2_9373_09d1c47de7d1.slice/crio-1a90d53d2de26450f3221ce9a072584777b7261584301f0df383387bb3f2c026 WatchSource:0}: Error finding container 1a90d53d2de26450f3221ce9a072584777b7261584301f0df383387bb3f2c026: Status 404 returned error can't find the container with id 1a90d53d2de26450f3221ce9a072584777b7261584301f0df383387bb3f2c026 Nov 03 20:59:39.249667 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:39.249628 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed16b66-kdsks" event={"ID":"57fea02a-046f-4cd2-9373-09d1c47de7d1","Type":"ContainerStarted","Data":"1a90d53d2de26450f3221ce9a072584777b7261584301f0df383387bb3f2c026"} Nov 03 20:59:39.375457 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:39.374895 4305 patch_prober.go:29] interesting pod/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 20:59:39.375457 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:39.374961 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz" podUID="f5b60f69-9404-49bd-a072-d04f9b4ccb19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 20:59:40.284016 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:40.283911 4305 generic.go:356] "Generic (PLEG): container finished" podID="57fea02a-046f-4cd2-9373-09d1c47de7d1" containerID="1ac02e123f643510b8d45f00a609596448b3e421ee7b35d987619ef6897a2627" exitCode=0 Nov 03 20:59:40.284718 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:40.283989 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed16b66-kdsks" event={"ID":"57fea02a-046f-4cd2-9373-09d1c47de7d1","Type":"ContainerDied","Data":"1ac02e123f643510b8d45f00a609596448b3e421ee7b35d987619ef6897a2627"} Nov 03 20:59:40.376927 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:40.376890 4305 patch_prober.go:29] interesting pod/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 20:59:40.377135 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:40.376955 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz" podUID="f5b60f69-9404-49bd-a072-d04f9b4ccb19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 20:59:41.379637 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:41.379598 4305 patch_prober.go:29] interesting pod/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 20:59:41.380193 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:41.379661 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz" podUID="f5b60f69-9404-49bd-a072-d04f9b4ccb19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 20:59:41.500922 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:41.500896 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed16b66-kdsks" Nov 03 20:59:41.590670 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:41.590585 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57fea02a-046f-4cd2-9373-09d1c47de7d1-kube-api-access-6qvwv\" (UniqueName: \"kubernetes.io/projected/57fea02a-046f-4cd2-9373-09d1c47de7d1-kube-api-access-6qvwv\") pod \"57fea02a-046f-4cd2-9373-09d1c47de7d1\" (UID: \"57fea02a-046f-4cd2-9373-09d1c47de7d1\") " Nov 03 20:59:41.590812 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:41.590777 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57fea02a-046f-4cd2-9373-09d1c47de7d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57fea02a-046f-4cd2-9373-09d1c47de7d1-maas-api-openshift-ca\") pod \"57fea02a-046f-4cd2-9373-09d1c47de7d1\" (UID: \"57fea02a-046f-4cd2-9373-09d1c47de7d1\") " Nov 03 20:59:41.590984 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:41.590967 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57fea02a-046f-4cd2-9373-09d1c47de7d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57fea02a-046f-4cd2-9373-09d1c47de7d1-maas-api-xks-ca\") pod \"57fea02a-046f-4cd2-9373-09d1c47de7d1\" (UID: \"57fea02a-046f-4cd2-9373-09d1c47de7d1\") " Nov 03 20:59:41.591126 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:41.591094 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57fea02a-046f-4cd2-9373-09d1c47de7d1-maas-api-openshift-ca" pod "57fea02a-046f-4cd2-9373-09d1c47de7d1" (UID: "57fea02a-046f-4cd2-9373-09d1c47de7d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:59:41.591523 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:41.591504 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57fea02a-046f-4cd2-9373-09d1c47de7d1-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:59:41.593015 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:41.592990 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57fea02a-046f-4cd2-9373-09d1c47de7d1-kube-api-access-6qvwv" pod "57fea02a-046f-4cd2-9373-09d1c47de7d1" (UID: "57fea02a-046f-4cd2-9373-09d1c47de7d1"). InnerVolumeSpecName "kube-api-access-6qvwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:59:41.593116 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:41.593021 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57fea02a-046f-4cd2-9373-09d1c47de7d1-maas-api-xks-ca" pod "57fea02a-046f-4cd2-9373-09d1c47de7d1" (UID: "57fea02a-046f-4cd2-9373-09d1c47de7d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:59:41.693313 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:41.693274 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57fea02a-046f-4cd2-9373-09d1c47de7d1-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:59:41.693313 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:41.693315 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qvwv\" (UniqueName: \"kubernetes.io/projected/57fea02a-046f-4cd2-9373-09d1c47de7d1-kube-api-access-6qvwv\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 20:59:42.301702 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:42.301671 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed16b66-kdsks" Nov 03 20:59:42.301913 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:42.301675 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed16b66-kdsks" event={"ID":"57fea02a-046f-4cd2-9373-09d1c47de7d1","Type":"ContainerDied","Data":"1a90d53d2de26450f3221ce9a072584777b7261584301f0df383387bb3f2c026"} Nov 03 20:59:42.301913 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:42.301807 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a90d53d2de26450f3221ce9a072584777b7261584301f0df383387bb3f2c026" Nov 03 20:59:42.377057 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:42.377005 4305 patch_prober.go:29] interesting pod/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 20:59:42.377256 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:42.377089 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz" podUID="f5b60f69-9404-49bd-a072-d04f9b4ccb19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 20:59:43.374459 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:43.373868 4305 patch_prober.go:29] interesting pod/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 20:59:43.374459 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:43.373934 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz" podUID="f5b60f69-9404-49bd-a072-d04f9b4ccb19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 20:59:44.374118 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:44.374080 4305 patch_prober.go:29] interesting pod/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 20:59:44.374283 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:44.374138 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz" podUID="f5b60f69-9404-49bd-a072-d04f9b4ccb19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 20:59:45.466784 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:45.466752 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz" Nov 03 20:59:45.467766 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:45.467741 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz" Nov 03 20:59:45.468658 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:45.468638 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz" Nov 03 20:59:46.322333 ip-10-0-128-175 kubenswrapper[4305]: E1103 20:59:46.322299 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 20:59:46.571332 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:46.571298 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8"] Nov 03 20:59:46.571983 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:46.571967 4305 cpu_manager.go:414] "containerMap: removing container" podUID="57fea02a-046f-4cd2-9373-09d1c47de7d1" containerName="revoke-keys" Nov 03 20:59:46.571983 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:46.571983 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57fea02a-046f-4cd2-9373-09d1c47de7d1" containerName="revoke-keys" Nov 03 20:59:46.572118 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:46.572110 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="57fea02a-046f-4cd2-9373-09d1c47de7d1" containerName="revoke-keys" Nov 03 20:59:46.575772 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:46.575718 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" Nov 03 20:59:46.663141 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:46.663103 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63a704b4-51ce-46c3-8007-34b308aff332-tmp-dir\") pod \"distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8\" (UID: \"63a704b4-51ce-46c3-8007-34b308aff332\") " pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" Nov 03 20:59:46.663141 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:46.663145 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63a704b4-51ce-46c3-8007-34b308aff332-tls-certs\") pod \"distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8\" (UID: \"63a704b4-51ce-46c3-8007-34b308aff332\") " pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" Nov 03 20:59:46.663380 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:46.663269 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63a704b4-51ce-46c3-8007-34b308aff332-dshm\") pod \"distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8\" (UID: \"63a704b4-51ce-46c3-8007-34b308aff332\") " pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" Nov 03 20:59:46.663380 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:46.663373 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63a704b4-51ce-46c3-8007-34b308aff332-model-cache\") pod \"distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8\" (UID: \"63a704b4-51ce-46c3-8007-34b308aff332\") " pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" Nov 03 20:59:46.663566 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:46.663543 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9x5q\" (UniqueName: \"kubernetes.io/projected/63a704b4-51ce-46c3-8007-34b308aff332-kube-api-access-x9x5q\") pod \"distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8\" (UID: \"63a704b4-51ce-46c3-8007-34b308aff332\") " pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" Nov 03 20:59:46.663695 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:46.663673 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63a704b4-51ce-46c3-8007-34b308aff332-home\") pod \"distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8\" (UID: \"63a704b4-51ce-46c3-8007-34b308aff332\") " pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" Nov 03 20:59:46.690240 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:46.690189 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-e22abf\"/\"distinct-2-w5-e22abf-kserve-self-signed-certs\"" Nov 03 20:59:46.695148 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:46.695119 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8"] Nov 03 20:59:46.765840 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:46.765805 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63a704b4-51ce-46c3-8007-34b308aff332-dshm\") pod \"distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8\" (UID: \"63a704b4-51ce-46c3-8007-34b308aff332\") " pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" Nov 03 20:59:46.766044 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:46.765962 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63a704b4-51ce-46c3-8007-34b308aff332-model-cache\") pod \"distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8\" (UID: \"63a704b4-51ce-46c3-8007-34b308aff332\") " pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" Nov 03 20:59:46.766044 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:46.766029 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9x5q\" (UniqueName: \"kubernetes.io/projected/63a704b4-51ce-46c3-8007-34b308aff332-kube-api-access-x9x5q\") pod \"distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8\" (UID: \"63a704b4-51ce-46c3-8007-34b308aff332\") " pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" Nov 03 20:59:46.766170 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:46.766151 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63a704b4-51ce-46c3-8007-34b308aff332-home\") pod \"distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8\" (UID: \"63a704b4-51ce-46c3-8007-34b308aff332\") " pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" Nov 03 20:59:46.766225 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:46.766195 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63a704b4-51ce-46c3-8007-34b308aff332-tmp-dir\") pod \"distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8\" (UID: \"63a704b4-51ce-46c3-8007-34b308aff332\") " pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" Nov 03 20:59:46.766284 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:46.766229 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63a704b4-51ce-46c3-8007-34b308aff332-tls-certs\") pod \"distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8\" (UID: \"63a704b4-51ce-46c3-8007-34b308aff332\") " pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" Nov 03 20:59:46.766392 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:46.766369 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63a704b4-51ce-46c3-8007-34b308aff332-model-cache\") pod \"distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8\" (UID: \"63a704b4-51ce-46c3-8007-34b308aff332\") " pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" Nov 03 20:59:46.766622 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:46.766601 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63a704b4-51ce-46c3-8007-34b308aff332-home\") pod \"distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8\" (UID: \"63a704b4-51ce-46c3-8007-34b308aff332\") " pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" Nov 03 20:59:46.766912 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:46.766886 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63a704b4-51ce-46c3-8007-34b308aff332-tmp-dir\") pod \"distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8\" (UID: \"63a704b4-51ce-46c3-8007-34b308aff332\") " pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" Nov 03 20:59:46.768414 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:46.768392 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63a704b4-51ce-46c3-8007-34b308aff332-dshm\") pod \"distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8\" (UID: \"63a704b4-51ce-46c3-8007-34b308aff332\") " pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" Nov 03 20:59:46.769148 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:46.769129 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63a704b4-51ce-46c3-8007-34b308aff332-tls-certs\") pod \"distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8\" (UID: \"63a704b4-51ce-46c3-8007-34b308aff332\") " pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" Nov 03 20:59:46.831691 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:46.831612 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9x5q\" (UniqueName: \"kubernetes.io/projected/63a704b4-51ce-46c3-8007-34b308aff332-kube-api-access-x9x5q\") pod \"distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8\" (UID: \"63a704b4-51ce-46c3-8007-34b308aff332\") " pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" Nov 03 20:59:46.890055 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:46.890025 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" Nov 03 20:59:47.279168 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:47.279142 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8"] Nov 03 20:59:47.279299 ip-10-0-128-175 kubenswrapper[4305]: W1103 20:59:47.279231 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod63a704b4_51ce_46c3_8007_34b308aff332.slice/crio-d3778305d329f7a06c9bec3b2b8eb69e06bfd6a597d952f3f096af0d78d107ff WatchSource:0}: Error finding container d3778305d329f7a06c9bec3b2b8eb69e06bfd6a597d952f3f096af0d78d107ff: Status 404 returned error can't find the container with id d3778305d329f7a06c9bec3b2b8eb69e06bfd6a597d952f3f096af0d78d107ff Nov 03 20:59:47.353976 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:47.353949 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" event={"ID":"63a704b4-51ce-46c3-8007-34b308aff332","Type":"ContainerStarted","Data":"d3778305d329f7a06c9bec3b2b8eb69e06bfd6a597d952f3f096af0d78d107ff"} Nov 03 20:59:48.362941 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:48.362899 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" event={"ID":"63a704b4-51ce-46c3-8007-34b308aff332","Type":"ContainerStarted","Data":"cc6f011d0e33a6a978e37978709653548475ee42d7d839f30aa10f110a7a4a46"} Nov 03 20:59:48.479194 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:48.479119 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" podStartSLOduration=2.479102635 podStartE2EDuration="2.479102635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:59:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:59:48.478729616 +0000 UTC m=+1223.704049416" watchObservedRunningTime="2026-11-03 20:59:48.479102635 +0000 UTC m=+1223.704422425" Nov 03 20:59:48.896229 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:48.896196 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" Nov 03 20:59:53.061854 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:53.061814 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d0c9705-qv5tg"] Nov 03 20:59:53.108428 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:53.108395 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d0c9705-qv5tg"] Nov 03 20:59:54.896399 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:54.896365 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de93cd6a-03d9-4458-8fbb-1315f51890e0" path="/var/lib/kubelet/pods/de93cd6a-03d9-4458-8fbb-1315f51890e0/volumes" Nov 03 20:59:56.896291 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:56.896261 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" Nov 03 20:59:56.902904 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:56.902883 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" Nov 03 20:59:57.440737 ip-10-0-128-175 kubenswrapper[4305]: I1103 20:59:57.440704 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" Nov 03 21:00:00.170946 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:00.170911 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-12a423-29895660-2m8zf"] Nov 03 21:00:00.176337 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:00.176318 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-12a423-29895660-2m8zf" Nov 03 21:00:00.189472 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:00.189426 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca4642-29895660-d4bs5"] Nov 03 21:00:00.193685 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:00.193668 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca4642-29895660-d4bs5" Nov 03 21:00:00.207149 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:00.207128 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-499187-29895660-clk7p"] Nov 03 21:00:00.211487 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:00.211467 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-499187-29895660-clk7p" Nov 03 21:00:00.221670 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:00.221644 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895660-lr9xr"] Nov 03 21:00:00.226381 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:00.226362 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-lr9xr" Nov 03 21:00:00.232917 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:00.232764 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-89bb58-29895660-42drl"] Nov 03 21:00:00.236114 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:00.236095 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:00:00.237065 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:00.237048 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-89bb58-29895660-42drl" Nov 03 21:00:00.252253 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:00.252227 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77vtd\"" Nov 03 21:00:00.278930 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:00.278898 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-12a423-29895660-2m8zf"] Nov 03 21:00:00.285758 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:00.285729 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca4642-29895660-d4bs5"] Nov 03 21:00:00.304611 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:00.304574 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ca13043-ae14-4af0-9fca-f3b845fb7981-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-499187-29895660-clk7p\" (UID: \"6ca13043-ae14-4af0-9fca-f3b845fb7981\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-499187-29895660-clk7p" Nov 03 21:00:00.304776 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:00.304759 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab595c24-ac7b-42b1-8123-e9b6fa020bc0-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895660-lr9xr\" (UID: \"ab595c24-ac7b-42b1-8123-e9b6fa020bc0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-lr9xr" Nov 03 21:00:00.304837 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:00.304820 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18ce9b2e-c1e9-47e4-b29b-f2b272fef364-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ca4642-29895660-d4bs5\" (UID: \"18ce9b2e-c1e9-47e4-b29b-f2b272fef364\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca4642-29895660-d4bs5" Nov 03 21:00:00.305070 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:00.305048 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9pf2\" (UniqueName: \"kubernetes.io/projected/b0501e93-a801-41df-90ae-016dbcbfa61b-kube-api-access-c9pf2\") pod \"maas-api-key-cleanup-e2e-worker-w6-12a423-29895660-2m8zf\" (UID: \"b0501e93-a801-41df-90ae-016dbcbfa61b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-12a423-29895660-2m8zf" Nov 03 21:00:00.305139 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:00.305092 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klntw\" (UniqueName: \"kubernetes.io/projected/6ca13043-ae14-4af0-9fca-f3b845fb7981-kube-api-access-klntw\") pod \"maas-api-key-cleanup-e2e-worker-w0-499187-29895660-clk7p\" (UID: \"6ca13043-ae14-4af0-9fca-f3b845fb7981\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-499187-29895660-clk7p" Nov 03 21:00:00.305139 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:00.305118 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlxtt\" (UniqueName: \"kubernetes.io/projected/1c7fa470-f7fd-44e9-8003-a1701d279c2e-kube-api-access-dlxtt\") pod \"maas-api-key-cleanup-e2e-worker-w3-89bb58-29895660-42drl\" (UID: \"1c7fa470-f7fd-44e9-8003-a1701d279c2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-89bb58-29895660-42drl" Nov 03 21:00:00.305276 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:00.305251 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c7fa470-f7fd-44e9-8003-a1701d279c2e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-89bb58-29895660-42drl\" (UID: \"1c7fa470-f7fd-44e9-8003-a1701d279c2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-89bb58-29895660-42drl" Nov 03 21:00:00.305311 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:00.305300 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbv2g\" (UniqueName: \"kubernetes.io/projected/18ce9b2e-c1e9-47e4-b29b-f2b272fef364-kube-api-access-tbv2g\") pod \"maas-api-key-cleanup-e2e-worker-w4-ca4642-29895660-d4bs5\" (UID: \"18ce9b2e-c1e9-47e4-b29b-f2b272fef364\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca4642-29895660-d4bs5" Nov 03 21:00:00.305384 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:00.305363 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0501e93-a801-41df-90ae-016dbcbfa61b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-12a423-29895660-2m8zf\" (UID: \"b0501e93-a801-41df-90ae-016dbcbfa61b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-12a423-29895660-2m8zf" Nov 03 21:00:00.305522 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:00.305503 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdmkm\" (UniqueName: \"kubernetes.io/projected/ab595c24-ac7b-42b1-8123-e9b6fa020bc0-kube-api-access-mdmkm\") pod \"maas-api-key-cleanup-29895660-lr9xr\" (UID: \"ab595c24-ac7b-42b1-8123-e9b6fa020bc0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-lr9xr" Nov 03 21:00:00.332863 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:00.332831 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:00:00.334120 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:00.334091 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895660-lr9xr"] Nov 03 21:00:00.339298 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:00.339273 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-499187-29895660-clk7p"] Nov 03 21:00:00.343691 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:00.343667 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-89bb58-29895660-42drl"] Nov 03 21:00:00.407266 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:00.407234 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9pf2\" (UniqueName: \"kubernetes.io/projected/b0501e93-a801-41df-90ae-016dbcbfa61b-kube-api-access-c9pf2\") pod \"maas-api-key-cleanup-e2e-worker-w6-12a423-29895660-2m8zf\" (UID: \"b0501e93-a801-41df-90ae-016dbcbfa61b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-12a423-29895660-2m8zf" Nov 03 21:00:00.407266 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:00.407274 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klntw\" (UniqueName: \"kubernetes.io/projected/6ca13043-ae14-4af0-9fca-f3b845fb7981-kube-api-access-klntw\") pod \"maas-api-key-cleanup-e2e-worker-w0-499187-29895660-clk7p\" (UID: \"6ca13043-ae14-4af0-9fca-f3b845fb7981\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-499187-29895660-clk7p" Nov 03 21:00:00.407564 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:00.407298 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlxtt\" (UniqueName: \"kubernetes.io/projected/1c7fa470-f7fd-44e9-8003-a1701d279c2e-kube-api-access-dlxtt\") pod \"maas-api-key-cleanup-e2e-worker-w3-89bb58-29895660-42drl\" (UID: \"1c7fa470-f7fd-44e9-8003-a1701d279c2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-89bb58-29895660-42drl" Nov 03 21:00:00.407564 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:00.407352 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c7fa470-f7fd-44e9-8003-a1701d279c2e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-89bb58-29895660-42drl\" (UID: \"1c7fa470-f7fd-44e9-8003-a1701d279c2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-89bb58-29895660-42drl" Nov 03 21:00:00.407564 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:00.407382 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbv2g\" (UniqueName: \"kubernetes.io/projected/18ce9b2e-c1e9-47e4-b29b-f2b272fef364-kube-api-access-tbv2g\") pod \"maas-api-key-cleanup-e2e-worker-w4-ca4642-29895660-d4bs5\" (UID: \"18ce9b2e-c1e9-47e4-b29b-f2b272fef364\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca4642-29895660-d4bs5" Nov 03 21:00:00.407564 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:00.407399 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0501e93-a801-41df-90ae-016dbcbfa61b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-12a423-29895660-2m8zf\" (UID: \"b0501e93-a801-41df-90ae-016dbcbfa61b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-12a423-29895660-2m8zf" Nov 03 21:00:00.407564 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:00.407420 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdmkm\" (UniqueName: \"kubernetes.io/projected/ab595c24-ac7b-42b1-8123-e9b6fa020bc0-kube-api-access-mdmkm\") pod \"maas-api-key-cleanup-29895660-lr9xr\" (UID: \"ab595c24-ac7b-42b1-8123-e9b6fa020bc0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-lr9xr" Nov 03 21:00:00.407564 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:00.407498 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ca13043-ae14-4af0-9fca-f3b845fb7981-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-499187-29895660-clk7p\" (UID: \"6ca13043-ae14-4af0-9fca-f3b845fb7981\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-499187-29895660-clk7p" Nov 03 21:00:00.407870 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:00.407611 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab595c24-ac7b-42b1-8123-e9b6fa020bc0-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895660-lr9xr\" (UID: \"ab595c24-ac7b-42b1-8123-e9b6fa020bc0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-lr9xr" Nov 03 21:00:00.407870 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:00.407694 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18ce9b2e-c1e9-47e4-b29b-f2b272fef364-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ca4642-29895660-d4bs5\" (UID: \"18ce9b2e-c1e9-47e4-b29b-f2b272fef364\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca4642-29895660-d4bs5" Nov 03 21:00:00.408302 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:00.408283 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0501e93-a801-41df-90ae-016dbcbfa61b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-12a423-29895660-2m8zf\" (UID: \"b0501e93-a801-41df-90ae-016dbcbfa61b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-12a423-29895660-2m8zf" Nov 03 21:00:00.408698 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:00.408667 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c7fa470-f7fd-44e9-8003-a1701d279c2e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-89bb58-29895660-42drl\" (UID: \"1c7fa470-f7fd-44e9-8003-a1701d279c2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-89bb58-29895660-42drl" Nov 03 21:00:00.408806 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:00.408699 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab595c24-ac7b-42b1-8123-e9b6fa020bc0-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895660-lr9xr\" (UID: \"ab595c24-ac7b-42b1-8123-e9b6fa020bc0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-lr9xr" Nov 03 21:00:00.408806 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:00.408699 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ca13043-ae14-4af0-9fca-f3b845fb7981-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-499187-29895660-clk7p\" (UID: \"6ca13043-ae14-4af0-9fca-f3b845fb7981\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-499187-29895660-clk7p" Nov 03 21:00:00.408978 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:00.408958 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18ce9b2e-c1e9-47e4-b29b-f2b272fef364-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ca4642-29895660-d4bs5\" (UID: \"18ce9b2e-c1e9-47e4-b29b-f2b272fef364\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca4642-29895660-d4bs5" Nov 03 21:00:00.469075 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:00.468986 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlxtt\" (UniqueName: \"kubernetes.io/projected/1c7fa470-f7fd-44e9-8003-a1701d279c2e-kube-api-access-dlxtt\") pod \"maas-api-key-cleanup-e2e-worker-w3-89bb58-29895660-42drl\" (UID: \"1c7fa470-f7fd-44e9-8003-a1701d279c2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-89bb58-29895660-42drl" Nov 03 21:00:00.469255 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:00.469079 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9pf2\" (UniqueName: \"kubernetes.io/projected/b0501e93-a801-41df-90ae-016dbcbfa61b-kube-api-access-c9pf2\") pod \"maas-api-key-cleanup-e2e-worker-w6-12a423-29895660-2m8zf\" (UID: \"b0501e93-a801-41df-90ae-016dbcbfa61b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-12a423-29895660-2m8zf" Nov 03 21:00:00.469255 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:00.469151 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klntw\" (UniqueName: \"kubernetes.io/projected/6ca13043-ae14-4af0-9fca-f3b845fb7981-kube-api-access-klntw\") pod \"maas-api-key-cleanup-e2e-worker-w0-499187-29895660-clk7p\" (UID: \"6ca13043-ae14-4af0-9fca-f3b845fb7981\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-499187-29895660-clk7p" Nov 03 21:00:00.469385 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:00.469339 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdmkm\" (UniqueName: \"kubernetes.io/projected/ab595c24-ac7b-42b1-8123-e9b6fa020bc0-kube-api-access-mdmkm\") pod \"maas-api-key-cleanup-29895660-lr9xr\" (UID: \"ab595c24-ac7b-42b1-8123-e9b6fa020bc0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-lr9xr" Nov 03 21:00:00.469693 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:00.469674 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbv2g\" (UniqueName: \"kubernetes.io/projected/18ce9b2e-c1e9-47e4-b29b-f2b272fef364-kube-api-access-tbv2g\") pod \"maas-api-key-cleanup-e2e-worker-w4-ca4642-29895660-d4bs5\" (UID: \"18ce9b2e-c1e9-47e4-b29b-f2b272fef364\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca4642-29895660-d4bs5" Nov 03 21:00:00.491154 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:00.491120 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-12a423-29895660-2m8zf" Nov 03 21:00:00.507699 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:00.507670 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca4642-29895660-d4bs5" Nov 03 21:00:00.529506 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:00.528172 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-499187-29895660-clk7p" Nov 03 21:00:00.539761 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:00.539717 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-lr9xr" Nov 03 21:00:00.552423 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:00.552318 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-89bb58-29895660-42drl" Nov 03 21:00:00.888160 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:00:00.888116 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0501e93_a801_41df_90ae_016dbcbfa61b.slice/crio-53761e9c48aa3f54db5b6e1ed5f8ab6f8a6695245cb9c3f5d28e731f323b5dcd WatchSource:0}: Error finding container 53761e9c48aa3f54db5b6e1ed5f8ab6f8a6695245cb9c3f5d28e731f323b5dcd: Status 404 returned error can't find the container with id 53761e9c48aa3f54db5b6e1ed5f8ab6f8a6695245cb9c3f5d28e731f323b5dcd Nov 03 21:00:00.897412 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:00.897388 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-12a423-29895660-2m8zf"] Nov 03 21:00:00.996997 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:00.996955 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-89bb58-29895660-42drl"] Nov 03 21:00:00.997345 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:00:00.997323 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c7fa470_f7fd_44e9_8003_a1701d279c2e.slice/crio-e0dfebc4e7632925c215f6c18137b3bfcc31ade61aa155bcfcc889c0e8ccaeb6 WatchSource:0}: Error finding container e0dfebc4e7632925c215f6c18137b3bfcc31ade61aa155bcfcc889c0e8ccaeb6: Status 404 returned error can't find the container with id e0dfebc4e7632925c215f6c18137b3bfcc31ade61aa155bcfcc889c0e8ccaeb6 Nov 03 21:00:00.999330 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:00:00.999305 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ca13043_ae14_4af0_9fca_f3b845fb7981.slice/crio-c8718ddc58876425cd45a9aef269e64251db3d5af45accbd0ebefb6a10cef185 WatchSource:0}: Error finding container c8718ddc58876425cd45a9aef269e64251db3d5af45accbd0ebefb6a10cef185: Status 404 returned error can't find the container with id c8718ddc58876425cd45a9aef269e64251db3d5af45accbd0ebefb6a10cef185 Nov 03 21:00:01.002870 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:01.002849 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-499187-29895660-clk7p"] Nov 03 21:00:01.052701 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:01.052660 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895660-lr9xr"] Nov 03 21:00:01.115290 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:01.115262 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca4642-29895660-d4bs5"] Nov 03 21:00:01.156572 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:00:01.156536 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18ce9b2e_c1e9_47e4_b29b_f2b272fef364.slice/crio-d84f091bac5006b9ce95d9f661c10c55c7de1e4bf4d4c8c09befb662e595af3a WatchSource:0}: Error finding container d84f091bac5006b9ce95d9f661c10c55c7de1e4bf4d4c8c09befb662e595af3a: Status 404 returned error can't find the container with id d84f091bac5006b9ce95d9f661c10c55c7de1e4bf4d4c8c09befb662e595af3a Nov 03 21:00:01.463636 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:01.463339 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-12a423-29895660-2m8zf" event={"ID":"b0501e93-a801-41df-90ae-016dbcbfa61b","Type":"ContainerStarted","Data":"53761e9c48aa3f54db5b6e1ed5f8ab6f8a6695245cb9c3f5d28e731f323b5dcd"} Nov 03 21:00:01.465508 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:01.465475 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-lr9xr" event={"ID":"ab595c24-ac7b-42b1-8123-e9b6fa020bc0","Type":"ContainerStarted","Data":"f0a70d3b966cb9e4cfdf60f453f26a644e6244eb2198e5d7480bd787175439cf"} Nov 03 21:00:01.469470 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:01.467204 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-89bb58-29895660-42drl" event={"ID":"1c7fa470-f7fd-44e9-8003-a1701d279c2e","Type":"ContainerStarted","Data":"e0dfebc4e7632925c215f6c18137b3bfcc31ade61aa155bcfcc889c0e8ccaeb6"} Nov 03 21:00:01.469470 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:01.469304 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-499187-29895660-clk7p" event={"ID":"6ca13043-ae14-4af0-9fca-f3b845fb7981","Type":"ContainerStarted","Data":"c8718ddc58876425cd45a9aef269e64251db3d5af45accbd0ebefb6a10cef185"} Nov 03 21:00:01.477364 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:01.477340 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca4642-29895660-d4bs5" event={"ID":"18ce9b2e-c1e9-47e4-b29b-f2b272fef364","Type":"ContainerStarted","Data":"d84f091bac5006b9ce95d9f661c10c55c7de1e4bf4d4c8c09befb662e595af3a"} Nov 03 21:00:01.580326 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:01.580294 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:00:02.486709 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:02.486678 4305 generic.go:356] "Generic (PLEG): container finished" podID="1c7fa470-f7fd-44e9-8003-a1701d279c2e" containerID="0b832969899526a1eab4a0d59555aa5419bd2f9c8a054937424d938e0cb4935c" exitCode=60 Nov 03 21:00:02.487186 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:02.486754 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-89bb58-29895660-42drl" event={"ID":"1c7fa470-f7fd-44e9-8003-a1701d279c2e","Type":"ContainerDied","Data":"0b832969899526a1eab4a0d59555aa5419bd2f9c8a054937424d938e0cb4935c"} Nov 03 21:00:02.487186 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:02.487089 4305 scope.go:122] "RemoveContainer" containerID="0b832969899526a1eab4a0d59555aa5419bd2f9c8a054937424d938e0cb4935c" Nov 03 21:00:02.488574 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:02.488551 4305 generic.go:356] "Generic (PLEG): container finished" podID="6ca13043-ae14-4af0-9fca-f3b845fb7981" containerID="b7fe63054343b9da37a15239cce7832305dc81d97b57b8625f03d1ae76d1d382" exitCode=60 Nov 03 21:00:02.488675 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:02.488649 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-499187-29895660-clk7p" event={"ID":"6ca13043-ae14-4af0-9fca-f3b845fb7981","Type":"ContainerDied","Data":"b7fe63054343b9da37a15239cce7832305dc81d97b57b8625f03d1ae76d1d382"} Nov 03 21:00:02.489011 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:02.488996 4305 scope.go:122] "RemoveContainer" containerID="b7fe63054343b9da37a15239cce7832305dc81d97b57b8625f03d1ae76d1d382" Nov 03 21:00:02.491980 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:02.491958 4305 generic.go:356] "Generic (PLEG): container finished" podID="18ce9b2e-c1e9-47e4-b29b-f2b272fef364" containerID="03c028cc27cdea0e4edc5b25e164b5ab6d71a892be8c9b6c3fedbdb78fdf283e" exitCode=60 Nov 03 21:00:02.492183 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:02.492000 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca4642-29895660-d4bs5" event={"ID":"18ce9b2e-c1e9-47e4-b29b-f2b272fef364","Type":"ContainerDied","Data":"03c028cc27cdea0e4edc5b25e164b5ab6d71a892be8c9b6c3fedbdb78fdf283e"} Nov 03 21:00:02.492295 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:02.492245 4305 scope.go:122] "RemoveContainer" containerID="03c028cc27cdea0e4edc5b25e164b5ab6d71a892be8c9b6c3fedbdb78fdf283e" Nov 03 21:00:02.494270 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:02.494253 4305 generic.go:356] "Generic (PLEG): container finished" podID="b0501e93-a801-41df-90ae-016dbcbfa61b" containerID="f0b3a812a9251a66eeb896e1238c752f9c2143f6924647f59ad8f781fd07602b" exitCode=60 Nov 03 21:00:02.494376 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:02.494342 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-12a423-29895660-2m8zf" event={"ID":"b0501e93-a801-41df-90ae-016dbcbfa61b","Type":"ContainerDied","Data":"f0b3a812a9251a66eeb896e1238c752f9c2143f6924647f59ad8f781fd07602b"} Nov 03 21:00:02.494623 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:02.494604 4305 scope.go:122] "RemoveContainer" containerID="f0b3a812a9251a66eeb896e1238c752f9c2143f6924647f59ad8f781fd07602b" Nov 03 21:00:02.496136 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:02.496118 4305 generic.go:356] "Generic (PLEG): container finished" podID="ab595c24-ac7b-42b1-8123-e9b6fa020bc0" containerID="f263bfc6f30c40a2460aadac873cb148f009a9008a9cf47e26a23c243dc26a59" exitCode=60 Nov 03 21:00:02.496245 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:02.496203 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-lr9xr" event={"ID":"ab595c24-ac7b-42b1-8123-e9b6fa020bc0","Type":"ContainerDied","Data":"f263bfc6f30c40a2460aadac873cb148f009a9008a9cf47e26a23c243dc26a59"} Nov 03 21:00:02.496366 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:02.496353 4305 scope.go:122] "RemoveContainer" containerID="f263bfc6f30c40a2460aadac873cb148f009a9008a9cf47e26a23c243dc26a59" Nov 03 21:00:03.506814 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:03.506779 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-89bb58-29895660-42drl" event={"ID":"1c7fa470-f7fd-44e9-8003-a1701d279c2e","Type":"ContainerStarted","Data":"1b19540220aabbc36493d0cc98bf58407f974a872298dfadc9ffa10b6f760514"} Nov 03 21:00:03.509198 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:03.509149 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-499187-29895660-clk7p" event={"ID":"6ca13043-ae14-4af0-9fca-f3b845fb7981","Type":"ContainerStarted","Data":"19a865c6618fcd8d0c81577a4704876f6f69cf860398506f0a745187a6f626f0"} Nov 03 21:00:03.511487 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:03.511462 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca4642-29895660-d4bs5" event={"ID":"18ce9b2e-c1e9-47e4-b29b-f2b272fef364","Type":"ContainerStarted","Data":"4b8417e5fc361ed7f0a9cbff7622f7e4c4b514e26846eea704c8d93c4c15992b"} Nov 03 21:00:03.513711 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:03.513685 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-12a423-29895660-2m8zf" event={"ID":"b0501e93-a801-41df-90ae-016dbcbfa61b","Type":"ContainerStarted","Data":"31e55cb8b0929753023fc10183d99a4e82b2a40af51b4f10a2b38c122eccffeb"} Nov 03 21:00:03.515869 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:03.515848 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-lr9xr" event={"ID":"ab595c24-ac7b-42b1-8123-e9b6fa020bc0","Type":"ContainerStarted","Data":"c80f11d9985b9ae30bffcbc34b681142ccc4eff874b5085a07c0c20fb6e30f4c"} Nov 03 21:00:03.625689 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:03.625631 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-89bb58-29895660-42drl" podStartSLOduration=3.6256152310000003 podStartE2EDuration="3.625615231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:03.622567531 +0000 UTC m=+1238.847887321" watchObservedRunningTime="2026-11-03 21:00:03.625615231 +0000 UTC m=+1238.850935019" Nov 03 21:00:03.746869 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:03.746813 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca4642-29895660-d4bs5" podStartSLOduration=3.7467966280000002 podStartE2EDuration="3.746796628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:03.743068691 +0000 UTC m=+1238.968388501" watchObservedRunningTime="2026-11-03 21:00:03.746796628 +0000 UTC m=+1238.972116444" Nov 03 21:00:03.860028 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:03.859976 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-12a423-29895660-2m8zf" podStartSLOduration=3.8599602109999998 podStartE2EDuration="3.859960211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:03.857355383 +0000 UTC m=+1239.082675192" watchObservedRunningTime="2026-11-03 21:00:03.859960211 +0000 UTC m=+1239.085280003" Nov 03 21:00:03.976083 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:03.976028 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-lr9xr" podStartSLOduration=3.976011046 podStartE2EDuration="3.976011046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:03.971115691 +0000 UTC m=+1239.196435500" watchObservedRunningTime="2026-11-03 21:00:03.976011046 +0000 UTC m=+1239.201330836" Nov 03 21:00:04.087333 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:04.087230 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-499187-29895660-clk7p" podStartSLOduration=4.087215149 podStartE2EDuration="4.087215149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:04.085137867 +0000 UTC m=+1239.310457675" watchObservedRunningTime="2026-11-03 21:00:04.087215149 +0000 UTC m=+1239.312534938" Nov 03 21:00:04.526394 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:04.526358 4305 generic.go:356] "Generic (PLEG): container finished" podID="b0501e93-a801-41df-90ae-016dbcbfa61b" containerID="31e55cb8b0929753023fc10183d99a4e82b2a40af51b4f10a2b38c122eccffeb" exitCode=60 Nov 03 21:00:04.526901 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:04.526412 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-12a423-29895660-2m8zf" event={"ID":"b0501e93-a801-41df-90ae-016dbcbfa61b","Type":"ContainerDied","Data":"31e55cb8b0929753023fc10183d99a4e82b2a40af51b4f10a2b38c122eccffeb"} Nov 03 21:00:04.526901 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:04.526506 4305 scope.go:122] "RemoveContainer" containerID="f0b3a812a9251a66eeb896e1238c752f9c2143f6924647f59ad8f781fd07602b" Nov 03 21:00:04.527268 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:04.527242 4305 scope.go:122] "RemoveContainer" containerID="31e55cb8b0929753023fc10183d99a4e82b2a40af51b4f10a2b38c122eccffeb" Nov 03 21:00:04.528032 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:00:04.528009 4305 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-12a423-29895660-2m8zf_odh-ai-gateway-infra(b0501e93-a801-41df-90ae-016dbcbfa61b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-12a423-29895660-2m8zf" podUID="b0501e93-a801-41df-90ae-016dbcbfa61b" Nov 03 21:00:04.528792 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:04.528772 4305 generic.go:356] "Generic (PLEG): container finished" podID="ab595c24-ac7b-42b1-8123-e9b6fa020bc0" containerID="c80f11d9985b9ae30bffcbc34b681142ccc4eff874b5085a07c0c20fb6e30f4c" exitCode=60 Nov 03 21:00:04.528962 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:04.528946 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-lr9xr" event={"ID":"ab595c24-ac7b-42b1-8123-e9b6fa020bc0","Type":"ContainerDied","Data":"c80f11d9985b9ae30bffcbc34b681142ccc4eff874b5085a07c0c20fb6e30f4c"} Nov 03 21:00:04.529368 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:04.529354 4305 scope.go:122] "RemoveContainer" containerID="c80f11d9985b9ae30bffcbc34b681142ccc4eff874b5085a07c0c20fb6e30f4c" Nov 03 21:00:04.529742 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:00:04.529720 4305 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895660-lr9xr_odh-ai-gateway-infra(ab595c24-ac7b-42b1-8123-e9b6fa020bc0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-lr9xr" podUID="ab595c24-ac7b-42b1-8123-e9b6fa020bc0" Nov 03 21:00:04.532386 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:04.532360 4305 generic.go:356] "Generic (PLEG): container finished" podID="1c7fa470-f7fd-44e9-8003-a1701d279c2e" containerID="1b19540220aabbc36493d0cc98bf58407f974a872298dfadc9ffa10b6f760514" exitCode=60 Nov 03 21:00:04.532506 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:04.532401 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-89bb58-29895660-42drl" event={"ID":"1c7fa470-f7fd-44e9-8003-a1701d279c2e","Type":"ContainerDied","Data":"1b19540220aabbc36493d0cc98bf58407f974a872298dfadc9ffa10b6f760514"} Nov 03 21:00:04.532814 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:04.532786 4305 scope.go:122] "RemoveContainer" containerID="1b19540220aabbc36493d0cc98bf58407f974a872298dfadc9ffa10b6f760514" Nov 03 21:00:04.533084 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:00:04.533068 4305 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-89bb58-29895660-42drl_odh-ai-gateway-infra(1c7fa470-f7fd-44e9-8003-a1701d279c2e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-89bb58-29895660-42drl" podUID="1c7fa470-f7fd-44e9-8003-a1701d279c2e" Nov 03 21:00:04.537253 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:04.537233 4305 generic.go:356] "Generic (PLEG): container finished" podID="6ca13043-ae14-4af0-9fca-f3b845fb7981" containerID="19a865c6618fcd8d0c81577a4704876f6f69cf860398506f0a745187a6f626f0" exitCode=60 Nov 03 21:00:04.537368 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:04.537325 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-499187-29895660-clk7p" event={"ID":"6ca13043-ae14-4af0-9fca-f3b845fb7981","Type":"ContainerDied","Data":"19a865c6618fcd8d0c81577a4704876f6f69cf860398506f0a745187a6f626f0"} Nov 03 21:00:04.537606 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:04.537590 4305 scope.go:122] "RemoveContainer" containerID="19a865c6618fcd8d0c81577a4704876f6f69cf860398506f0a745187a6f626f0" Nov 03 21:00:04.537917 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:00:04.537894 4305 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-499187-29895660-clk7p_odh-ai-gateway-infra(6ca13043-ae14-4af0-9fca-f3b845fb7981)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-499187-29895660-clk7p" podUID="6ca13043-ae14-4af0-9fca-f3b845fb7981" Nov 03 21:00:04.539325 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:04.539305 4305 generic.go:356] "Generic (PLEG): container finished" podID="18ce9b2e-c1e9-47e4-b29b-f2b272fef364" containerID="4b8417e5fc361ed7f0a9cbff7622f7e4c4b514e26846eea704c8d93c4c15992b" exitCode=60 Nov 03 21:00:04.539427 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:04.539354 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca4642-29895660-d4bs5" event={"ID":"18ce9b2e-c1e9-47e4-b29b-f2b272fef364","Type":"ContainerDied","Data":"4b8417e5fc361ed7f0a9cbff7622f7e4c4b514e26846eea704c8d93c4c15992b"} Nov 03 21:00:04.539619 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:04.539604 4305 scope.go:122] "RemoveContainer" containerID="4b8417e5fc361ed7f0a9cbff7622f7e4c4b514e26846eea704c8d93c4c15992b" Nov 03 21:00:04.539867 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:00:04.539850 4305 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ca4642-29895660-d4bs5_odh-ai-gateway-infra(18ce9b2e-c1e9-47e4-b29b-f2b272fef364)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca4642-29895660-d4bs5" podUID="18ce9b2e-c1e9-47e4-b29b-f2b272fef364" Nov 03 21:00:04.542714 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:04.542697 4305 scope.go:122] "RemoveContainer" containerID="f263bfc6f30c40a2460aadac873cb148f009a9008a9cf47e26a23c243dc26a59" Nov 03 21:00:04.557553 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:04.557528 4305 scope.go:122] "RemoveContainer" containerID="0b832969899526a1eab4a0d59555aa5419bd2f9c8a054937424d938e0cb4935c" Nov 03 21:00:04.571893 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:04.571866 4305 scope.go:122] "RemoveContainer" containerID="b7fe63054343b9da37a15239cce7832305dc81d97b57b8625f03d1ae76d1d382" Nov 03 21:00:04.586502 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:04.586300 4305 scope.go:122] "RemoveContainer" containerID="03c028cc27cdea0e4edc5b25e164b5ab6d71a892be8c9b6c3fedbdb78fdf283e" Nov 03 21:00:05.550875 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:05.550845 4305 scope.go:122] "RemoveContainer" containerID="c80f11d9985b9ae30bffcbc34b681142ccc4eff874b5085a07c0c20fb6e30f4c" Nov 03 21:00:05.551362 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:00:05.551219 4305 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895660-lr9xr_odh-ai-gateway-infra(ab595c24-ac7b-42b1-8123-e9b6fa020bc0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-lr9xr" podUID="ab595c24-ac7b-42b1-8123-e9b6fa020bc0" Nov 03 21:00:05.552799 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:05.552777 4305 scope.go:122] "RemoveContainer" containerID="1b19540220aabbc36493d0cc98bf58407f974a872298dfadc9ffa10b6f760514" Nov 03 21:00:05.553115 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:00:05.553098 4305 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-89bb58-29895660-42drl_odh-ai-gateway-infra(1c7fa470-f7fd-44e9-8003-a1701d279c2e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-89bb58-29895660-42drl" podUID="1c7fa470-f7fd-44e9-8003-a1701d279c2e" Nov 03 21:00:05.554457 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:05.554426 4305 scope.go:122] "RemoveContainer" containerID="19a865c6618fcd8d0c81577a4704876f6f69cf860398506f0a745187a6f626f0" Nov 03 21:00:05.554729 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:00:05.554708 4305 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-499187-29895660-clk7p_odh-ai-gateway-infra(6ca13043-ae14-4af0-9fca-f3b845fb7981)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-499187-29895660-clk7p" podUID="6ca13043-ae14-4af0-9fca-f3b845fb7981" Nov 03 21:00:05.556073 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:05.556055 4305 scope.go:122] "RemoveContainer" containerID="4b8417e5fc361ed7f0a9cbff7622f7e4c4b514e26846eea704c8d93c4c15992b" Nov 03 21:00:05.556323 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:00:05.556299 4305 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ca4642-29895660-d4bs5_odh-ai-gateway-infra(18ce9b2e-c1e9-47e4-b29b-f2b272fef364)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca4642-29895660-d4bs5" podUID="18ce9b2e-c1e9-47e4-b29b-f2b272fef364" Nov 03 21:00:05.557731 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:05.557714 4305 scope.go:122] "RemoveContainer" containerID="31e55cb8b0929753023fc10183d99a4e82b2a40af51b4f10a2b38c122eccffeb" Nov 03 21:00:05.557968 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:00:05.557950 4305 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-12a423-29895660-2m8zf_odh-ai-gateway-infra(b0501e93-a801-41df-90ae-016dbcbfa61b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-12a423-29895660-2m8zf" podUID="b0501e93-a801-41df-90ae-016dbcbfa61b" Nov 03 21:00:08.160405 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:08.160367 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4"] Nov 03 21:00:08.165980 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:08.165958 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" Nov 03 21:00:08.246422 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:08.246386 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be06bc96-808a-46e6-9872-01baf1a11a8d-tmp-dir\") pod \"distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4\" (UID: \"be06bc96-808a-46e6-9872-01baf1a11a8d\") " pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" Nov 03 21:00:08.246619 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:08.246453 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be06bc96-808a-46e6-9872-01baf1a11a8d-model-cache\") pod \"distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4\" (UID: \"be06bc96-808a-46e6-9872-01baf1a11a8d\") " pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" Nov 03 21:00:08.246619 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:08.246521 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be06bc96-808a-46e6-9872-01baf1a11a8d-dshm\") pod \"distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4\" (UID: \"be06bc96-808a-46e6-9872-01baf1a11a8d\") " pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" Nov 03 21:00:08.246619 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:08.246555 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8v4r\" (UniqueName: \"kubernetes.io/projected/be06bc96-808a-46e6-9872-01baf1a11a8d-kube-api-access-v8v4r\") pod \"distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4\" (UID: \"be06bc96-808a-46e6-9872-01baf1a11a8d\") " pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" Nov 03 21:00:08.246818 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:08.246793 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be06bc96-808a-46e6-9872-01baf1a11a8d-tls-certs\") pod \"distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4\" (UID: \"be06bc96-808a-46e6-9872-01baf1a11a8d\") " pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" Nov 03 21:00:08.246876 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:08.246839 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be06bc96-808a-46e6-9872-01baf1a11a8d-home\") pod \"distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4\" (UID: \"be06bc96-808a-46e6-9872-01baf1a11a8d\") " pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" Nov 03 21:00:08.268095 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:08.268054 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ca4642\"/\"distinct-2-w4-ca4642-kserve-self-signed-certs\"" Nov 03 21:00:08.273538 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:08.273508 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4"] Nov 03 21:00:08.348013 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:08.347981 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be06bc96-808a-46e6-9872-01baf1a11a8d-tls-certs\") pod \"distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4\" (UID: \"be06bc96-808a-46e6-9872-01baf1a11a8d\") " pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" Nov 03 21:00:08.348013 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:08.348019 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be06bc96-808a-46e6-9872-01baf1a11a8d-home\") pod \"distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4\" (UID: \"be06bc96-808a-46e6-9872-01baf1a11a8d\") " pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" Nov 03 21:00:08.348284 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:08.348075 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be06bc96-808a-46e6-9872-01baf1a11a8d-tmp-dir\") pod \"distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4\" (UID: \"be06bc96-808a-46e6-9872-01baf1a11a8d\") " pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" Nov 03 21:00:08.348284 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:08.348095 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be06bc96-808a-46e6-9872-01baf1a11a8d-model-cache\") pod \"distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4\" (UID: \"be06bc96-808a-46e6-9872-01baf1a11a8d\") " pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" Nov 03 21:00:08.348284 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:08.348122 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be06bc96-808a-46e6-9872-01baf1a11a8d-dshm\") pod \"distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4\" (UID: \"be06bc96-808a-46e6-9872-01baf1a11a8d\") " pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" Nov 03 21:00:08.348284 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:08.348150 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8v4r\" (UniqueName: \"kubernetes.io/projected/be06bc96-808a-46e6-9872-01baf1a11a8d-kube-api-access-v8v4r\") pod \"distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4\" (UID: \"be06bc96-808a-46e6-9872-01baf1a11a8d\") " pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" Nov 03 21:00:08.348630 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:08.348601 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be06bc96-808a-46e6-9872-01baf1a11a8d-tmp-dir\") pod \"distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4\" (UID: \"be06bc96-808a-46e6-9872-01baf1a11a8d\") " pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" Nov 03 21:00:08.348745 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:08.348601 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be06bc96-808a-46e6-9872-01baf1a11a8d-home\") pod \"distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4\" (UID: \"be06bc96-808a-46e6-9872-01baf1a11a8d\") " pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" Nov 03 21:00:08.348745 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:08.348658 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be06bc96-808a-46e6-9872-01baf1a11a8d-model-cache\") pod \"distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4\" (UID: \"be06bc96-808a-46e6-9872-01baf1a11a8d\") " pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" Nov 03 21:00:08.350525 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:08.350497 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be06bc96-808a-46e6-9872-01baf1a11a8d-dshm\") pod \"distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4\" (UID: \"be06bc96-808a-46e6-9872-01baf1a11a8d\") " pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" Nov 03 21:00:08.350723 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:08.350693 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be06bc96-808a-46e6-9872-01baf1a11a8d-tls-certs\") pod \"distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4\" (UID: \"be06bc96-808a-46e6-9872-01baf1a11a8d\") " pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" Nov 03 21:00:08.409199 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:08.409168 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8v4r\" (UniqueName: \"kubernetes.io/projected/be06bc96-808a-46e6-9872-01baf1a11a8d-kube-api-access-v8v4r\") pod \"distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4\" (UID: \"be06bc96-808a-46e6-9872-01baf1a11a8d\") " pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" Nov 03 21:00:08.479384 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:08.479293 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" Nov 03 21:00:08.889795 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:00:08.889760 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbe06bc96_808a_46e6_9872_01baf1a11a8d.slice/crio-f5db05a87ceedb9042a912e51d8fb8b358c7f3b6572e3c9387b7b599e9ccb56e WatchSource:0}: Error finding container f5db05a87ceedb9042a912e51d8fb8b358c7f3b6572e3c9387b7b599e9ccb56e: Status 404 returned error can't find the container with id f5db05a87ceedb9042a912e51d8fb8b358c7f3b6572e3c9387b7b599e9ccb56e Nov 03 21:00:08.898337 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:08.898309 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4"] Nov 03 21:00:09.623809 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:09.623776 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" event={"ID":"be06bc96-808a-46e6-9872-01baf1a11a8d","Type":"ContainerStarted","Data":"ece75c0b5cd7e0b25238ecbce9b0648aac6247002dfbfd6590167e97b7b6791e"} Nov 03 21:00:09.624259 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:09.623823 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" event={"ID":"be06bc96-808a-46e6-9872-01baf1a11a8d","Type":"ContainerStarted","Data":"f5db05a87ceedb9042a912e51d8fb8b358c7f3b6572e3c9387b7b599e9ccb56e"} Nov 03 21:00:09.745368 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:09.745308 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" podStartSLOduration=1.74529359 podStartE2EDuration="1.74529359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:09.74127777 +0000 UTC m=+1244.966597553" watchObservedRunningTime="2026-11-03 21:00:09.74529359 +0000 UTC m=+1244.970613380" Nov 03 21:00:10.479741 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:10.479697 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" Nov 03 21:00:14.896199 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:14.896167 4305 scope.go:122] "RemoveContainer" containerID="19a865c6618fcd8d0c81577a4704876f6f69cf860398506f0a745187a6f626f0" Nov 03 21:00:14.896716 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:14.896291 4305 scope.go:122] "RemoveContainer" containerID="31e55cb8b0929753023fc10183d99a4e82b2a40af51b4f10a2b38c122eccffeb" Nov 03 21:00:15.670100 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:15.670069 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-12a423-29895660-2m8zf" event={"ID":"b0501e93-a801-41df-90ae-016dbcbfa61b","Type":"ContainerStarted","Data":"7d60fb3ff2b23ca925eca08e8918ed75d4aa869828d2a10c8ef8685e4564141b"} Nov 03 21:00:15.672165 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:15.672139 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-499187-29895660-clk7p" event={"ID":"6ca13043-ae14-4af0-9fca-f3b845fb7981","Type":"ContainerStarted","Data":"8402254a046a30df7ccbb01e7bbcd0f3a9b13177388bbc82fb5c1938526a9461"} Nov 03 21:00:15.888070 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:15.888037 4305 scope.go:122] "RemoveContainer" containerID="1b19540220aabbc36493d0cc98bf58407f974a872298dfadc9ffa10b6f760514" Nov 03 21:00:16.323253 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:00:16.323213 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:00:16.681330 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:16.681291 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-89bb58-29895660-42drl" event={"ID":"1c7fa470-f7fd-44e9-8003-a1701d279c2e","Type":"ContainerStarted","Data":"e869abc5fe2095298237720be5f0beca9ac404a22e11abcdde7a1433c93324c9"} Nov 03 21:00:16.683225 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:16.683199 4305 generic.go:356] "Generic (PLEG): container finished" podID="6ca13043-ae14-4af0-9fca-f3b845fb7981" containerID="8402254a046a30df7ccbb01e7bbcd0f3a9b13177388bbc82fb5c1938526a9461" exitCode=60 Nov 03 21:00:16.683344 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:16.683268 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-499187-29895660-clk7p" event={"ID":"6ca13043-ae14-4af0-9fca-f3b845fb7981","Type":"ContainerDied","Data":"8402254a046a30df7ccbb01e7bbcd0f3a9b13177388bbc82fb5c1938526a9461"} Nov 03 21:00:16.683344 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:16.683314 4305 scope.go:122] "RemoveContainer" containerID="19a865c6618fcd8d0c81577a4704876f6f69cf860398506f0a745187a6f626f0" Nov 03 21:00:16.683684 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:16.683666 4305 scope.go:122] "RemoveContainer" containerID="8402254a046a30df7ccbb01e7bbcd0f3a9b13177388bbc82fb5c1938526a9461" Nov 03 21:00:16.684008 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:00:16.683985 4305 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-499187-29895660-clk7p_odh-ai-gateway-infra(6ca13043-ae14-4af0-9fca-f3b845fb7981)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-499187-29895660-clk7p" podUID="6ca13043-ae14-4af0-9fca-f3b845fb7981" Nov 03 21:00:16.685451 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:16.685412 4305 generic.go:356] "Generic (PLEG): container finished" podID="b0501e93-a801-41df-90ae-016dbcbfa61b" containerID="7d60fb3ff2b23ca925eca08e8918ed75d4aa869828d2a10c8ef8685e4564141b" exitCode=60 Nov 03 21:00:16.685529 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:16.685453 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-12a423-29895660-2m8zf" event={"ID":"b0501e93-a801-41df-90ae-016dbcbfa61b","Type":"ContainerDied","Data":"7d60fb3ff2b23ca925eca08e8918ed75d4aa869828d2a10c8ef8685e4564141b"} Nov 03 21:00:16.685760 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:16.685746 4305 scope.go:122] "RemoveContainer" containerID="7d60fb3ff2b23ca925eca08e8918ed75d4aa869828d2a10c8ef8685e4564141b" Nov 03 21:00:16.685991 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:00:16.685975 4305 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-12a423-29895660-2m8zf_odh-ai-gateway-infra(b0501e93-a801-41df-90ae-016dbcbfa61b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-12a423-29895660-2m8zf" podUID="b0501e93-a801-41df-90ae-016dbcbfa61b" Nov 03 21:00:16.697227 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:16.697208 4305 scope.go:122] "RemoveContainer" containerID="31e55cb8b0929753023fc10183d99a4e82b2a40af51b4f10a2b38c122eccffeb" Nov 03 21:00:16.797292 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:16.797260 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-12a423-29895660-2m8zf"] Nov 03 21:00:16.899898 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:16.899863 4305 scope.go:122] "RemoveContainer" containerID="c80f11d9985b9ae30bffcbc34b681142ccc4eff874b5085a07c0c20fb6e30f4c" Nov 03 21:00:16.913887 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:16.913853 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-499187-29895660-clk7p"] Nov 03 21:00:17.695130 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:17.695096 4305 generic.go:356] "Generic (PLEG): container finished" podID="1c7fa470-f7fd-44e9-8003-a1701d279c2e" containerID="e869abc5fe2095298237720be5f0beca9ac404a22e11abcdde7a1433c93324c9" exitCode=60 Nov 03 21:00:17.695635 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:17.695164 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-89bb58-29895660-42drl" event={"ID":"1c7fa470-f7fd-44e9-8003-a1701d279c2e","Type":"ContainerDied","Data":"e869abc5fe2095298237720be5f0beca9ac404a22e11abcdde7a1433c93324c9"} Nov 03 21:00:17.695635 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:17.695253 4305 scope.go:122] "RemoveContainer" containerID="1b19540220aabbc36493d0cc98bf58407f974a872298dfadc9ffa10b6f760514" Nov 03 21:00:17.695738 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:17.695668 4305 scope.go:122] "RemoveContainer" containerID="e869abc5fe2095298237720be5f0beca9ac404a22e11abcdde7a1433c93324c9" Nov 03 21:00:17.696053 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:00:17.696024 4305 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-89bb58-29895660-42drl_odh-ai-gateway-infra(1c7fa470-f7fd-44e9-8003-a1701d279c2e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-89bb58-29895660-42drl" podUID="1c7fa470-f7fd-44e9-8003-a1701d279c2e" Nov 03 21:00:17.701226 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:17.701202 4305 generic.go:356] "Generic (PLEG): container finished" podID="ab595c24-ac7b-42b1-8123-e9b6fa020bc0" containerID="7917d7c693c18dafe146ab5a49ba6915fe7fc64e8113fd1ba8a926713400fcb9" exitCode=60 Nov 03 21:00:17.701340 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:17.701283 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-lr9xr" event={"ID":"ab595c24-ac7b-42b1-8123-e9b6fa020bc0","Type":"ContainerDied","Data":"7917d7c693c18dafe146ab5a49ba6915fe7fc64e8113fd1ba8a926713400fcb9"} Nov 03 21:00:17.701907 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:17.701888 4305 scope.go:122] "RemoveContainer" containerID="7917d7c693c18dafe146ab5a49ba6915fe7fc64e8113fd1ba8a926713400fcb9" Nov 03 21:00:17.702317 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:00:17.702295 4305 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895660-lr9xr_odh-ai-gateway-infra(ab595c24-ac7b-42b1-8123-e9b6fa020bc0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-lr9xr" podUID="ab595c24-ac7b-42b1-8123-e9b6fa020bc0" Nov 03 21:00:17.710305 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:17.710283 4305 scope.go:122] "RemoveContainer" containerID="c80f11d9985b9ae30bffcbc34b681142ccc4eff874b5085a07c0c20fb6e30f4c" Nov 03 21:00:17.809603 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:17.809569 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-89bb58-29895660-42drl"] Nov 03 21:00:17.882358 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:17.882283 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-12a423-29895660-2m8zf" Nov 03 21:00:17.889292 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:17.889256 4305 scope.go:122] "RemoveContainer" containerID="4b8417e5fc361ed7f0a9cbff7622f7e4c4b514e26846eea704c8d93c4c15992b" Nov 03 21:00:17.957149 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:17.957122 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-499187-29895660-clk7p" Nov 03 21:00:17.997717 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:17.997686 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0501e93-a801-41df-90ae-016dbcbfa61b-kube-api-access-c9pf2\" (UniqueName: \"kubernetes.io/projected/b0501e93-a801-41df-90ae-016dbcbfa61b-kube-api-access-c9pf2\") pod \"b0501e93-a801-41df-90ae-016dbcbfa61b\" (UID: \"b0501e93-a801-41df-90ae-016dbcbfa61b\") " Nov 03 21:00:17.997938 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:17.997918 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0501e93-a801-41df-90ae-016dbcbfa61b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0501e93-a801-41df-90ae-016dbcbfa61b-maas-api-service-ca\") pod \"b0501e93-a801-41df-90ae-016dbcbfa61b\" (UID: \"b0501e93-a801-41df-90ae-016dbcbfa61b\") " Nov 03 21:00:17.998288 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:17.998251 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0501e93-a801-41df-90ae-016dbcbfa61b-maas-api-service-ca" pod "b0501e93-a801-41df-90ae-016dbcbfa61b" (UID: "b0501e93-a801-41df-90ae-016dbcbfa61b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:00:17.999351 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:17.999328 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0501e93-a801-41df-90ae-016dbcbfa61b-maas-api-service-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:00:18.052094 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:18.052058 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0501e93-a801-41df-90ae-016dbcbfa61b-kube-api-access-c9pf2" pod "b0501e93-a801-41df-90ae-016dbcbfa61b" (UID: "b0501e93-a801-41df-90ae-016dbcbfa61b"). InnerVolumeSpecName "kube-api-access-c9pf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:18.100697 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:18.100654 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ca13043-ae14-4af0-9fca-f3b845fb7981-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ca13043-ae14-4af0-9fca-f3b845fb7981-maas-api-service-ca\") pod \"6ca13043-ae14-4af0-9fca-f3b845fb7981\" (UID: \"6ca13043-ae14-4af0-9fca-f3b845fb7981\") " Nov 03 21:00:18.100915 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:18.100792 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ca13043-ae14-4af0-9fca-f3b845fb7981-kube-api-access-klntw\" (UniqueName: \"kubernetes.io/projected/6ca13043-ae14-4af0-9fca-f3b845fb7981-kube-api-access-klntw\") pod \"6ca13043-ae14-4af0-9fca-f3b845fb7981\" (UID: \"6ca13043-ae14-4af0-9fca-f3b845fb7981\") " Nov 03 21:00:18.101057 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:18.101022 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ca13043-ae14-4af0-9fca-f3b845fb7981-maas-api-service-ca" pod "6ca13043-ae14-4af0-9fca-f3b845fb7981" (UID: "6ca13043-ae14-4af0-9fca-f3b845fb7981"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:00:18.101412 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:18.101389 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9pf2\" (UniqueName: \"kubernetes.io/projected/b0501e93-a801-41df-90ae-016dbcbfa61b-kube-api-access-c9pf2\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:00:18.101559 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:18.101415 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ca13043-ae14-4af0-9fca-f3b845fb7981-maas-api-service-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:00:18.103130 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:18.103107 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ca13043-ae14-4af0-9fca-f3b845fb7981-kube-api-access-klntw" pod "6ca13043-ae14-4af0-9fca-f3b845fb7981" (UID: "6ca13043-ae14-4af0-9fca-f3b845fb7981"). InnerVolumeSpecName "kube-api-access-klntw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:18.202547 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:18.202420 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klntw\" (UniqueName: \"kubernetes.io/projected/6ca13043-ae14-4af0-9fca-f3b845fb7981-kube-api-access-klntw\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:00:18.480386 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:18.480356 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" Nov 03 21:00:18.493381 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:18.493354 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" Nov 03 21:00:18.711092 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:18.710997 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca4642-29895660-d4bs5" event={"ID":"18ce9b2e-c1e9-47e4-b29b-f2b272fef364","Type":"ContainerStarted","Data":"bfb8f91c41d12bb0963195b7496e616319089725acfd090638338c3e85d3a8e6"} Nov 03 21:00:18.712719 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:18.712700 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-12a423-29895660-2m8zf" Nov 03 21:00:18.712857 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:18.712744 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-12a423-29895660-2m8zf" event={"ID":"b0501e93-a801-41df-90ae-016dbcbfa61b","Type":"ContainerDied","Data":"53761e9c48aa3f54db5b6e1ed5f8ab6f8a6695245cb9c3f5d28e731f323b5dcd"} Nov 03 21:00:18.712857 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:18.712813 4305 scope.go:122] "RemoveContainer" containerID="7d60fb3ff2b23ca925eca08e8918ed75d4aa869828d2a10c8ef8685e4564141b" Nov 03 21:00:18.720015 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:18.719989 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-499187-29895660-clk7p" Nov 03 21:00:18.720348 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:18.720268 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-499187-29895660-clk7p" event={"ID":"6ca13043-ae14-4af0-9fca-f3b845fb7981","Type":"ContainerDied","Data":"c8718ddc58876425cd45a9aef269e64251db3d5af45accbd0ebefb6a10cef185"} Nov 03 21:00:18.728802 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:18.728782 4305 scope.go:122] "RemoveContainer" containerID="8402254a046a30df7ccbb01e7bbcd0f3a9b13177388bbc82fb5c1938526a9461" Nov 03 21:00:18.734151 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:18.734126 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" Nov 03 21:00:18.886782 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:18.886757 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-89bb58-29895660-42drl" Nov 03 21:00:18.940591 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:18.940555 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-12a423-29895660-2m8zf"] Nov 03 21:00:18.946356 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:18.946322 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-12a423-29895660-2m8zf"] Nov 03 21:00:19.012329 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:19.012289 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c7fa470-f7fd-44e9-8003-a1701d279c2e-kube-api-access-dlxtt\" (UniqueName: \"kubernetes.io/projected/1c7fa470-f7fd-44e9-8003-a1701d279c2e-kube-api-access-dlxtt\") pod \"1c7fa470-f7fd-44e9-8003-a1701d279c2e\" (UID: \"1c7fa470-f7fd-44e9-8003-a1701d279c2e\") " Nov 03 21:00:19.012588 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:19.012473 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c7fa470-f7fd-44e9-8003-a1701d279c2e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c7fa470-f7fd-44e9-8003-a1701d279c2e-maas-api-service-ca\") pod \"1c7fa470-f7fd-44e9-8003-a1701d279c2e\" (UID: \"1c7fa470-f7fd-44e9-8003-a1701d279c2e\") " Nov 03 21:00:19.012799 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:19.012780 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c7fa470-f7fd-44e9-8003-a1701d279c2e-maas-api-service-ca" pod "1c7fa470-f7fd-44e9-8003-a1701d279c2e" (UID: "1c7fa470-f7fd-44e9-8003-a1701d279c2e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:00:19.013057 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:19.013029 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c7fa470-f7fd-44e9-8003-a1701d279c2e-maas-api-service-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:00:19.014647 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:19.014629 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c7fa470-f7fd-44e9-8003-a1701d279c2e-kube-api-access-dlxtt" pod "1c7fa470-f7fd-44e9-8003-a1701d279c2e" (UID: "1c7fa470-f7fd-44e9-8003-a1701d279c2e"). InnerVolumeSpecName "kube-api-access-dlxtt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:19.055644 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:19.055593 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-499187-29895660-clk7p"] Nov 03 21:00:19.060422 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:19.060391 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-499187-29895660-clk7p"] Nov 03 21:00:19.075239 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:19.075196 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895660-lr9xr"] Nov 03 21:00:19.114710 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:19.114675 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlxtt\" (UniqueName: \"kubernetes.io/projected/1c7fa470-f7fd-44e9-8003-a1701d279c2e-kube-api-access-dlxtt\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:00:19.216229 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:19.216200 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-lr9xr" Nov 03 21:00:19.317238 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:19.317147 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab595c24-ac7b-42b1-8123-e9b6fa020bc0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab595c24-ac7b-42b1-8123-e9b6fa020bc0-maas-api-service-ca\") pod \"ab595c24-ac7b-42b1-8123-e9b6fa020bc0\" (UID: \"ab595c24-ac7b-42b1-8123-e9b6fa020bc0\") " Nov 03 21:00:19.317238 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:19.317201 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab595c24-ac7b-42b1-8123-e9b6fa020bc0-kube-api-access-mdmkm\" (UniqueName: \"kubernetes.io/projected/ab595c24-ac7b-42b1-8123-e9b6fa020bc0-kube-api-access-mdmkm\") pod \"ab595c24-ac7b-42b1-8123-e9b6fa020bc0\" (UID: \"ab595c24-ac7b-42b1-8123-e9b6fa020bc0\") " Nov 03 21:00:19.317604 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:19.317581 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab595c24-ac7b-42b1-8123-e9b6fa020bc0-maas-api-service-ca" pod "ab595c24-ac7b-42b1-8123-e9b6fa020bc0" (UID: "ab595c24-ac7b-42b1-8123-e9b6fa020bc0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:00:19.317804 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:19.317790 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab595c24-ac7b-42b1-8123-e9b6fa020bc0-maas-api-service-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:00:19.319494 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:19.319472 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab595c24-ac7b-42b1-8123-e9b6fa020bc0-kube-api-access-mdmkm" pod "ab595c24-ac7b-42b1-8123-e9b6fa020bc0" (UID: "ab595c24-ac7b-42b1-8123-e9b6fa020bc0"). InnerVolumeSpecName "kube-api-access-mdmkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:19.419074 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:19.419037 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdmkm\" (UniqueName: \"kubernetes.io/projected/ab595c24-ac7b-42b1-8123-e9b6fa020bc0-kube-api-access-mdmkm\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:00:19.728923 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:19.728896 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-lr9xr" Nov 03 21:00:19.729475 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:19.728936 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-lr9xr" event={"ID":"ab595c24-ac7b-42b1-8123-e9b6fa020bc0","Type":"ContainerDied","Data":"f0a70d3b966cb9e4cfdf60f453f26a644e6244eb2198e5d7480bd787175439cf"} Nov 03 21:00:19.729475 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:19.728999 4305 scope.go:122] "RemoveContainer" containerID="7917d7c693c18dafe146ab5a49ba6915fe7fc64e8113fd1ba8a926713400fcb9" Nov 03 21:00:19.730673 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:19.730650 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-89bb58-29895660-42drl" event={"ID":"1c7fa470-f7fd-44e9-8003-a1701d279c2e","Type":"ContainerDied","Data":"e0dfebc4e7632925c215f6c18137b3bfcc31ade61aa155bcfcc889c0e8ccaeb6"} Nov 03 21:00:19.730673 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:19.730676 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-89bb58-29895660-42drl" Nov 03 21:00:19.734267 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:19.734243 4305 generic.go:356] "Generic (PLEG): container finished" podID="18ce9b2e-c1e9-47e4-b29b-f2b272fef364" containerID="bfb8f91c41d12bb0963195b7496e616319089725acfd090638338c3e85d3a8e6" exitCode=60 Nov 03 21:00:19.734387 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:19.734299 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca4642-29895660-d4bs5" event={"ID":"18ce9b2e-c1e9-47e4-b29b-f2b272fef364","Type":"ContainerDied","Data":"bfb8f91c41d12bb0963195b7496e616319089725acfd090638338c3e85d3a8e6"} Nov 03 21:00:19.734780 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:19.734758 4305 scope.go:122] "RemoveContainer" containerID="bfb8f91c41d12bb0963195b7496e616319089725acfd090638338c3e85d3a8e6" Nov 03 21:00:19.735143 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:00:19.735115 4305 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ca4642-29895660-d4bs5_odh-ai-gateway-infra(18ce9b2e-c1e9-47e4-b29b-f2b272fef364)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca4642-29895660-d4bs5" podUID="18ce9b2e-c1e9-47e4-b29b-f2b272fef364" Nov 03 21:00:19.742344 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:19.742325 4305 scope.go:122] "RemoveContainer" containerID="e869abc5fe2095298237720be5f0beca9ac404a22e11abcdde7a1433c93324c9" Nov 03 21:00:19.752841 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:19.752819 4305 scope.go:122] "RemoveContainer" containerID="4b8417e5fc361ed7f0a9cbff7622f7e4c4b514e26846eea704c8d93c4c15992b" Nov 03 21:00:19.857404 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:19.857367 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca4642-29895660-d4bs5"] Nov 03 21:00:20.044423 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:20.044333 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-89bb58-29895660-42drl"] Nov 03 21:00:20.050870 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:20.050828 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-89bb58-29895660-42drl"] Nov 03 21:00:20.161220 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:20.161188 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895660-lr9xr"] Nov 03 21:00:20.166302 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:20.166272 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895660-lr9xr"] Nov 03 21:00:20.525798 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:20.525761 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac83ebbd-bxvgz"] Nov 03 21:00:20.526754 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:20.526732 4305 cpu_manager.go:414] "containerMap: removing container" podUID="1c7fa470-f7fd-44e9-8003-a1701d279c2e" containerName="cleanup" Nov 03 21:00:20.526825 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:20.526757 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c7fa470-f7fd-44e9-8003-a1701d279c2e" containerName="cleanup" Nov 03 21:00:20.526825 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:20.526773 4305 cpu_manager.go:414] "containerMap: removing container" podUID="ab595c24-ac7b-42b1-8123-e9b6fa020bc0" containerName="cleanup" Nov 03 21:00:20.526825 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:20.526780 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab595c24-ac7b-42b1-8123-e9b6fa020bc0" containerName="cleanup" Nov 03 21:00:20.526825 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:20.526794 4305 cpu_manager.go:414] "containerMap: removing container" podUID="b0501e93-a801-41df-90ae-016dbcbfa61b" containerName="cleanup" Nov 03 21:00:20.526825 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:20.526800 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0501e93-a801-41df-90ae-016dbcbfa61b" containerName="cleanup" Nov 03 21:00:20.526825 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:20.526813 4305 cpu_manager.go:414] "containerMap: removing container" podUID="6ca13043-ae14-4af0-9fca-f3b845fb7981" containerName="cleanup" Nov 03 21:00:20.526825 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:20.526820 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ca13043-ae14-4af0-9fca-f3b845fb7981" containerName="cleanup" Nov 03 21:00:20.527017 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:20.526852 4305 cpu_manager.go:414] "containerMap: removing container" podUID="b0501e93-a801-41df-90ae-016dbcbfa61b" containerName="cleanup" Nov 03 21:00:20.527017 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:20.526860 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0501e93-a801-41df-90ae-016dbcbfa61b" containerName="cleanup" Nov 03 21:00:20.527017 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:20.526877 4305 cpu_manager.go:414] "containerMap: removing container" podUID="6ca13043-ae14-4af0-9fca-f3b845fb7981" containerName="cleanup" Nov 03 21:00:20.527017 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:20.526883 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ca13043-ae14-4af0-9fca-f3b845fb7981" containerName="cleanup" Nov 03 21:00:20.527017 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:20.526893 4305 cpu_manager.go:414] "containerMap: removing container" podUID="1c7fa470-f7fd-44e9-8003-a1701d279c2e" containerName="cleanup" Nov 03 21:00:20.527017 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:20.526899 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c7fa470-f7fd-44e9-8003-a1701d279c2e" containerName="cleanup" Nov 03 21:00:20.527017 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:20.526924 4305 cpu_manager.go:414] "containerMap: removing container" podUID="b0501e93-a801-41df-90ae-016dbcbfa61b" containerName="cleanup" Nov 03 21:00:20.527017 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:20.526930 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0501e93-a801-41df-90ae-016dbcbfa61b" containerName="cleanup" Nov 03 21:00:20.527017 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:20.526940 4305 cpu_manager.go:414] "containerMap: removing container" podUID="ab595c24-ac7b-42b1-8123-e9b6fa020bc0" containerName="cleanup" Nov 03 21:00:20.527017 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:20.526946 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab595c24-ac7b-42b1-8123-e9b6fa020bc0" containerName="cleanup" Nov 03 21:00:20.527276 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:20.527095 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c7fa470-f7fd-44e9-8003-a1701d279c2e" containerName="cleanup" Nov 03 21:00:20.527276 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:20.527107 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab595c24-ac7b-42b1-8123-e9b6fa020bc0" containerName="cleanup" Nov 03 21:00:20.527276 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:20.527116 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0501e93-a801-41df-90ae-016dbcbfa61b" containerName="cleanup" Nov 03 21:00:20.527276 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:20.527125 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ca13043-ae14-4af0-9fca-f3b845fb7981" containerName="cleanup" Nov 03 21:00:20.527276 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:20.527138 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0501e93-a801-41df-90ae-016dbcbfa61b" containerName="cleanup" Nov 03 21:00:20.527276 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:20.527145 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab595c24-ac7b-42b1-8123-e9b6fa020bc0" containerName="cleanup" Nov 03 21:00:20.527276 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:20.527152 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ca13043-ae14-4af0-9fca-f3b845fb7981" containerName="cleanup" Nov 03 21:00:20.527276 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:20.527162 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ca13043-ae14-4af0-9fca-f3b845fb7981" containerName="cleanup" Nov 03 21:00:20.527276 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:20.527171 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c7fa470-f7fd-44e9-8003-a1701d279c2e" containerName="cleanup" Nov 03 21:00:20.527276 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:20.527180 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0501e93-a801-41df-90ae-016dbcbfa61b" containerName="cleanup" Nov 03 21:00:20.532251 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:20.532230 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac83ebbd-bxvgz" Nov 03 21:00:20.584333 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:20.584303 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:00:20.585312 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:20.585287 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac83ebbd-bxvgz"] Nov 03 21:00:20.642074 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:20.642037 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwz82\" (UniqueName: \"kubernetes.io/projected/8378fc38-80d3-4ca3-b413-c529bab97f09-kube-api-access-bwz82\") pod \"maas-api-revoke-sub-ac83ebbd-bxvgz\" (UID: \"8378fc38-80d3-4ca3-b413-c529bab97f09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac83ebbd-bxvgz" Nov 03 21:00:20.642261 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:20.642085 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8378fc38-80d3-4ca3-b413-c529bab97f09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac83ebbd-bxvgz\" (UID: \"8378fc38-80d3-4ca3-b413-c529bab97f09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac83ebbd-bxvgz" Nov 03 21:00:20.642261 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:20.642197 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8378fc38-80d3-4ca3-b413-c529bab97f09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac83ebbd-bxvgz\" (UID: \"8378fc38-80d3-4ca3-b413-c529bab97f09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac83ebbd-bxvgz" Nov 03 21:00:20.743464 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:20.743358 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwz82\" (UniqueName: \"kubernetes.io/projected/8378fc38-80d3-4ca3-b413-c529bab97f09-kube-api-access-bwz82\") pod \"maas-api-revoke-sub-ac83ebbd-bxvgz\" (UID: \"8378fc38-80d3-4ca3-b413-c529bab97f09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac83ebbd-bxvgz" Nov 03 21:00:20.743464 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:20.743405 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8378fc38-80d3-4ca3-b413-c529bab97f09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac83ebbd-bxvgz\" (UID: \"8378fc38-80d3-4ca3-b413-c529bab97f09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac83ebbd-bxvgz" Nov 03 21:00:20.744088 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:20.743497 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8378fc38-80d3-4ca3-b413-c529bab97f09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac83ebbd-bxvgz\" (UID: \"8378fc38-80d3-4ca3-b413-c529bab97f09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac83ebbd-bxvgz" Nov 03 21:00:20.744293 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:20.744262 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8378fc38-80d3-4ca3-b413-c529bab97f09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac83ebbd-bxvgz\" (UID: \"8378fc38-80d3-4ca3-b413-c529bab97f09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac83ebbd-bxvgz" Nov 03 21:00:20.746383 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:20.746354 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8378fc38-80d3-4ca3-b413-c529bab97f09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac83ebbd-bxvgz\" (UID: \"8378fc38-80d3-4ca3-b413-c529bab97f09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac83ebbd-bxvgz" Nov 03 21:00:20.804129 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:20.804098 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwz82\" (UniqueName: \"kubernetes.io/projected/8378fc38-80d3-4ca3-b413-c529bab97f09-kube-api-access-bwz82\") pod \"maas-api-revoke-sub-ac83ebbd-bxvgz\" (UID: \"8378fc38-80d3-4ca3-b413-c529bab97f09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac83ebbd-bxvgz" Nov 03 21:00:20.848588 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:20.848547 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac83ebbd-bxvgz" Nov 03 21:00:20.898627 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:20.898591 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c7fa470-f7fd-44e9-8003-a1701d279c2e" path="/var/lib/kubelet/pods/1c7fa470-f7fd-44e9-8003-a1701d279c2e/volumes" Nov 03 21:00:20.899140 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:20.899120 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ca13043-ae14-4af0-9fca-f3b845fb7981" path="/var/lib/kubelet/pods/6ca13043-ae14-4af0-9fca-f3b845fb7981/volumes" Nov 03 21:00:20.899616 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:20.899600 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab595c24-ac7b-42b1-8123-e9b6fa020bc0" path="/var/lib/kubelet/pods/ab595c24-ac7b-42b1-8123-e9b6fa020bc0/volumes" Nov 03 21:00:20.900444 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:20.900410 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0501e93-a801-41df-90ae-016dbcbfa61b" path="/var/lib/kubelet/pods/b0501e93-a801-41df-90ae-016dbcbfa61b/volumes" Nov 03 21:00:20.903885 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:20.903863 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca4642-29895660-d4bs5" Nov 03 21:00:21.049465 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:21.049338 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18ce9b2e-c1e9-47e4-b29b-f2b272fef364-kube-api-access-tbv2g\" (UniqueName: \"kubernetes.io/projected/18ce9b2e-c1e9-47e4-b29b-f2b272fef364-kube-api-access-tbv2g\") pod \"18ce9b2e-c1e9-47e4-b29b-f2b272fef364\" (UID: \"18ce9b2e-c1e9-47e4-b29b-f2b272fef364\") " Nov 03 21:00:21.049623 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:21.049589 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18ce9b2e-c1e9-47e4-b29b-f2b272fef364-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18ce9b2e-c1e9-47e4-b29b-f2b272fef364-maas-api-service-ca\") pod \"18ce9b2e-c1e9-47e4-b29b-f2b272fef364\" (UID: \"18ce9b2e-c1e9-47e4-b29b-f2b272fef364\") " Nov 03 21:00:21.049979 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:21.049952 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18ce9b2e-c1e9-47e4-b29b-f2b272fef364-maas-api-service-ca" pod "18ce9b2e-c1e9-47e4-b29b-f2b272fef364" (UID: "18ce9b2e-c1e9-47e4-b29b-f2b272fef364"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:00:21.050256 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:21.050235 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18ce9b2e-c1e9-47e4-b29b-f2b272fef364-maas-api-service-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:00:21.051379 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:21.051352 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-931ba160-mgltm"] Nov 03 21:00:21.054011 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:21.053982 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18ce9b2e-c1e9-47e4-b29b-f2b272fef364-kube-api-access-tbv2g" pod "18ce9b2e-c1e9-47e4-b29b-f2b272fef364" (UID: "18ce9b2e-c1e9-47e4-b29b-f2b272fef364"). InnerVolumeSpecName "kube-api-access-tbv2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:21.072599 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:21.072571 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-931ba160-mgltm"] Nov 03 21:00:21.152238 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:21.152197 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbv2g\" (UniqueName: \"kubernetes.io/projected/18ce9b2e-c1e9-47e4-b29b-f2b272fef364-kube-api-access-tbv2g\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:00:21.237608 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:00:21.237579 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8378fc38_80d3_4ca3_b413_c529bab97f09.slice/crio-efc187b58f1d47aab5770e42c7aada29b7a20dee1656d7a89a1e18673f347fe7 WatchSource:0}: Error finding container efc187b58f1d47aab5770e42c7aada29b7a20dee1656d7a89a1e18673f347fe7: Status 404 returned error can't find the container with id efc187b58f1d47aab5770e42c7aada29b7a20dee1656d7a89a1e18673f347fe7 Nov 03 21:00:21.238854 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:21.238830 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac83ebbd-bxvgz"] Nov 03 21:00:21.650989 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:21.650955 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl"] Nov 03 21:00:21.651686 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:21.651671 4305 cpu_manager.go:414] "containerMap: removing container" podUID="ab595c24-ac7b-42b1-8123-e9b6fa020bc0" containerName="cleanup" Nov 03 21:00:21.651734 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:21.651689 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab595c24-ac7b-42b1-8123-e9b6fa020bc0" containerName="cleanup" Nov 03 21:00:21.651734 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:21.651700 4305 cpu_manager.go:414] "containerMap: removing container" podUID="18ce9b2e-c1e9-47e4-b29b-f2b272fef364" containerName="cleanup" Nov 03 21:00:21.651734 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:21.651705 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18ce9b2e-c1e9-47e4-b29b-f2b272fef364" containerName="cleanup" Nov 03 21:00:21.651734 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:21.651728 4305 cpu_manager.go:414] "containerMap: removing container" podUID="1c7fa470-f7fd-44e9-8003-a1701d279c2e" containerName="cleanup" Nov 03 21:00:21.651734 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:21.651733 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c7fa470-f7fd-44e9-8003-a1701d279c2e" containerName="cleanup" Nov 03 21:00:21.651887 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:21.651761 4305 cpu_manager.go:414] "containerMap: removing container" podUID="6ca13043-ae14-4af0-9fca-f3b845fb7981" containerName="cleanup" Nov 03 21:00:21.651887 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:21.651765 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ca13043-ae14-4af0-9fca-f3b845fb7981" containerName="cleanup" Nov 03 21:00:21.651887 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:21.651777 4305 cpu_manager.go:414] "containerMap: removing container" podUID="18ce9b2e-c1e9-47e4-b29b-f2b272fef364" containerName="cleanup" Nov 03 21:00:21.651887 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:21.651782 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18ce9b2e-c1e9-47e4-b29b-f2b272fef364" containerName="cleanup" Nov 03 21:00:21.651887 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:21.651874 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="18ce9b2e-c1e9-47e4-b29b-f2b272fef364" containerName="cleanup" Nov 03 21:00:21.651887 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:21.651881 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="18ce9b2e-c1e9-47e4-b29b-f2b272fef364" containerName="cleanup" Nov 03 21:00:21.652077 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:21.651903 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c7fa470-f7fd-44e9-8003-a1701d279c2e" containerName="cleanup" Nov 03 21:00:21.652077 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:21.651918 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab595c24-ac7b-42b1-8123-e9b6fa020bc0" containerName="cleanup" Nov 03 21:00:21.657194 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:21.657170 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl" Nov 03 21:00:21.709731 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:21.709695 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-e22abf-openshift-default-dockercfg-8q8f9\"" Nov 03 21:00:21.722797 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:21.722714 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl"] Nov 03 21:00:21.761294 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:21.761260 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca4642-29895660-d4bs5" Nov 03 21:00:21.762050 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:21.761534 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca4642-29895660-d4bs5" event={"ID":"18ce9b2e-c1e9-47e4-b29b-f2b272fef364","Type":"ContainerDied","Data":"d84f091bac5006b9ce95d9f661c10c55c7de1e4bf4d4c8c09befb662e595af3a"} Nov 03 21:00:21.762050 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:21.761659 4305 scope.go:122] "RemoveContainer" containerID="bfb8f91c41d12bb0963195b7496e616319089725acfd090638338c3e85d3a8e6" Nov 03 21:00:21.765884 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:21.765855 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac83ebbd-bxvgz" event={"ID":"8378fc38-80d3-4ca3-b413-c529bab97f09","Type":"ContainerStarted","Data":"f3137d3590b804d50dcda25a28ebab28d2413e0900b7f35a75774eb8647a3104"} Nov 03 21:00:21.766056 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:21.765913 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac83ebbd-bxvgz" event={"ID":"8378fc38-80d3-4ca3-b413-c529bab97f09","Type":"ContainerStarted","Data":"efc187b58f1d47aab5770e42c7aada29b7a20dee1656d7a89a1e18673f347fe7"} Nov 03 21:00:21.769939 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:21.769910 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90f310bb-3fa1-404e-b893-6c49a2f760e1-istio-podinfo\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl\" (UID: \"90f310bb-3fa1-404e-b893-6c49a2f760e1\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl" Nov 03 21:00:21.770023 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:21.770000 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8gxx\" (UniqueName: \"kubernetes.io/projected/90f310bb-3fa1-404e-b893-6c49a2f760e1-kube-api-access-g8gxx\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl\" (UID: \"90f310bb-3fa1-404e-b893-6c49a2f760e1\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl" Nov 03 21:00:21.770080 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:21.770061 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f310bb-3fa1-404e-b893-6c49a2f760e1-workload-socket\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl\" (UID: \"90f310bb-3fa1-404e-b893-6c49a2f760e1\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl" Nov 03 21:00:21.770134 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:21.770093 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90f310bb-3fa1-404e-b893-6c49a2f760e1-istio-data\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl\" (UID: \"90f310bb-3fa1-404e-b893-6c49a2f760e1\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl" Nov 03 21:00:21.770134 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:21.770119 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90f310bb-3fa1-404e-b893-6c49a2f760e1-istio-token\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl\" (UID: \"90f310bb-3fa1-404e-b893-6c49a2f760e1\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl" Nov 03 21:00:21.770277 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:21.770259 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90f310bb-3fa1-404e-b893-6c49a2f760e1-istio-envoy\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl\" (UID: \"90f310bb-3fa1-404e-b893-6c49a2f760e1\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl" Nov 03 21:00:21.770346 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:21.770335 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f310bb-3fa1-404e-b893-6c49a2f760e1-credential-socket\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl\" (UID: \"90f310bb-3fa1-404e-b893-6c49a2f760e1\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl" Nov 03 21:00:21.770398 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:21.770384 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90f310bb-3fa1-404e-b893-6c49a2f760e1-workload-certs\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl\" (UID: \"90f310bb-3fa1-404e-b893-6c49a2f760e1\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl" Nov 03 21:00:21.770462 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:21.770413 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90f310bb-3fa1-404e-b893-6c49a2f760e1-istiod-ca-cert\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl\" (UID: \"90f310bb-3fa1-404e-b893-6c49a2f760e1\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl" Nov 03 21:00:21.871780 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:21.871749 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8gxx\" (UniqueName: \"kubernetes.io/projected/90f310bb-3fa1-404e-b893-6c49a2f760e1-kube-api-access-g8gxx\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl\" (UID: \"90f310bb-3fa1-404e-b893-6c49a2f760e1\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl" Nov 03 21:00:21.871780 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:21.871784 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f310bb-3fa1-404e-b893-6c49a2f760e1-workload-socket\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl\" (UID: \"90f310bb-3fa1-404e-b893-6c49a2f760e1\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl" Nov 03 21:00:21.872059 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:21.871813 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90f310bb-3fa1-404e-b893-6c49a2f760e1-istio-data\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl\" (UID: \"90f310bb-3fa1-404e-b893-6c49a2f760e1\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl" Nov 03 21:00:21.872059 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:21.871850 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90f310bb-3fa1-404e-b893-6c49a2f760e1-istio-token\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl\" (UID: \"90f310bb-3fa1-404e-b893-6c49a2f760e1\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl" Nov 03 21:00:21.872059 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:21.871975 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90f310bb-3fa1-404e-b893-6c49a2f760e1-istio-envoy\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl\" (UID: \"90f310bb-3fa1-404e-b893-6c49a2f760e1\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl" Nov 03 21:00:21.872208 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:21.872061 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f310bb-3fa1-404e-b893-6c49a2f760e1-credential-socket\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl\" (UID: \"90f310bb-3fa1-404e-b893-6c49a2f760e1\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl" Nov 03 21:00:21.872208 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:21.872108 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90f310bb-3fa1-404e-b893-6c49a2f760e1-workload-certs\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl\" (UID: \"90f310bb-3fa1-404e-b893-6c49a2f760e1\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl" Nov 03 21:00:21.872208 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:21.872132 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90f310bb-3fa1-404e-b893-6c49a2f760e1-istiod-ca-cert\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl\" (UID: \"90f310bb-3fa1-404e-b893-6c49a2f760e1\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl" Nov 03 21:00:21.872379 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:21.872253 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f310bb-3fa1-404e-b893-6c49a2f760e1-workload-socket\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl\" (UID: \"90f310bb-3fa1-404e-b893-6c49a2f760e1\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl" Nov 03 21:00:21.872379 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:21.872257 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90f310bb-3fa1-404e-b893-6c49a2f760e1-istio-data\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl\" (UID: \"90f310bb-3fa1-404e-b893-6c49a2f760e1\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl" Nov 03 21:00:21.872379 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:21.872317 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90f310bb-3fa1-404e-b893-6c49a2f760e1-istio-podinfo\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl\" (UID: \"90f310bb-3fa1-404e-b893-6c49a2f760e1\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl" Nov 03 21:00:21.872556 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:21.872426 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f310bb-3fa1-404e-b893-6c49a2f760e1-credential-socket\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl\" (UID: \"90f310bb-3fa1-404e-b893-6c49a2f760e1\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl" Nov 03 21:00:21.872556 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:21.872511 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90f310bb-3fa1-404e-b893-6c49a2f760e1-workload-certs\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl\" (UID: \"90f310bb-3fa1-404e-b893-6c49a2f760e1\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl" Nov 03 21:00:21.872798 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:21.872771 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90f310bb-3fa1-404e-b893-6c49a2f760e1-istiod-ca-cert\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl\" (UID: \"90f310bb-3fa1-404e-b893-6c49a2f760e1\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl" Nov 03 21:00:21.874593 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:21.874573 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90f310bb-3fa1-404e-b893-6c49a2f760e1-istio-envoy\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl\" (UID: \"90f310bb-3fa1-404e-b893-6c49a2f760e1\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl" Nov 03 21:00:21.874877 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:21.874848 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90f310bb-3fa1-404e-b893-6c49a2f760e1-istio-podinfo\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl\" (UID: \"90f310bb-3fa1-404e-b893-6c49a2f760e1\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl" Nov 03 21:00:21.894980 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:21.894951 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca4642-29895660-d4bs5"] Nov 03 21:00:21.903889 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:21.903868 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ca4642-29895660-d4bs5"] Nov 03 21:00:21.933570 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:21.933545 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90f310bb-3fa1-404e-b893-6c49a2f760e1-istio-token\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl\" (UID: \"90f310bb-3fa1-404e-b893-6c49a2f760e1\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl" Nov 03 21:00:21.933685 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:21.933564 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8gxx\" (UniqueName: \"kubernetes.io/projected/90f310bb-3fa1-404e-b893-6c49a2f760e1-kube-api-access-g8gxx\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl\" (UID: \"90f310bb-3fa1-404e-b893-6c49a2f760e1\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl" Nov 03 21:00:21.973512 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:21.973350 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl" Nov 03 21:00:22.022101 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:22.022040 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac83ebbd-bxvgz" podStartSLOduration=2.022023674 podStartE2EDuration="2.022023674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:22.020456432 +0000 UTC m=+1257.245776245" watchObservedRunningTime="2026-11-03 21:00:22.022023674 +0000 UTC m=+1257.247343466" Nov 03 21:00:22.522298 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:22.522216 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl"] Nov 03 21:00:22.525561 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:00:22.525530 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90f310bb_3fa1_404e_b893_6c49a2f760e1.slice/crio-14eb79710a42b2d5c289648361d3463058e95fd54c76c2aca7dda7d960d17af2 WatchSource:0}: Error finding container 14eb79710a42b2d5c289648361d3463058e95fd54c76c2aca7dda7d960d17af2: Status 404 returned error can't find the container with id 14eb79710a42b2d5c289648361d3463058e95fd54c76c2aca7dda7d960d17af2 Nov 03 21:00:22.529213 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:22.529155 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:00:22.529333 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:22.529303 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:00:22.529413 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:22.529390 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:00:22.813889 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:22.813794 4305 generic.go:356] "Generic (PLEG): container finished" podID="8378fc38-80d3-4ca3-b413-c529bab97f09" containerID="f3137d3590b804d50dcda25a28ebab28d2413e0900b7f35a75774eb8647a3104" exitCode=0 Nov 03 21:00:22.814547 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:22.814522 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac83ebbd-bxvgz" event={"ID":"8378fc38-80d3-4ca3-b413-c529bab97f09","Type":"ContainerDied","Data":"f3137d3590b804d50dcda25a28ebab28d2413e0900b7f35a75774eb8647a3104"} Nov 03 21:00:22.817344 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:22.817314 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl" event={"ID":"90f310bb-3fa1-404e-b893-6c49a2f760e1","Type":"ContainerStarted","Data":"b47203e5f72b4728f861c46b62dec7d58a6a684008e7cff760d6c95f8f20cde0"} Nov 03 21:00:22.817615 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:22.817585 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl" event={"ID":"90f310bb-3fa1-404e-b893-6c49a2f760e1","Type":"ContainerStarted","Data":"14eb79710a42b2d5c289648361d3463058e95fd54c76c2aca7dda7d960d17af2"} Nov 03 21:00:22.922406 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:22.916456 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18ce9b2e-c1e9-47e4-b29b-f2b272fef364" path="/var/lib/kubelet/pods/18ce9b2e-c1e9-47e4-b29b-f2b272fef364/volumes" Nov 03 21:00:22.922406 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:22.917176 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a10b3870-8ea5-4d05-8d81-c3061e45340e" path="/var/lib/kubelet/pods/a10b3870-8ea5-4d05-8d81-c3061e45340e/volumes" Nov 03 21:00:22.981593 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:22.981564 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl" Nov 03 21:00:23.099530 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:23.091407 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl" podStartSLOduration=2.091385122 podStartE2EDuration="2.091385122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:23.051376155 +0000 UTC m=+1258.276695983" watchObservedRunningTime="2026-11-03 21:00:23.091385122 +0000 UTC m=+1258.316704912" Nov 03 21:00:23.986464 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:23.981756 4305 patch_prober.go:29] interesting pod/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 03 21:00:23.986464 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:23.981815 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl" podUID="90f310bb-3fa1-404e-b893-6c49a2f760e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 03 21:00:24.263518 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:24.262541 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" podUID="7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:00:24.288051 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:24.287562 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" podUID="8ee1f028-8503-4034-9954-eb06fcf402e7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:00:24.320555 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:24.320510 4305 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" podUID="7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:00:24.507459 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:24.506532 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" podUID="7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 21:00:24.876517 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:24.869520 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac83ebbd-bxvgz" Nov 03 21:00:24.935025 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:24.933007 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8378fc38-80d3-4ca3-b413-c529bab97f09-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8378fc38-80d3-4ca3-b413-c529bab97f09-maas-api-xks-ca\") pod \"8378fc38-80d3-4ca3-b413-c529bab97f09\" (UID: \"8378fc38-80d3-4ca3-b413-c529bab97f09\") " Nov 03 21:00:24.935025 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:24.933193 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8378fc38-80d3-4ca3-b413-c529bab97f09-kube-api-access-bwz82\" (UniqueName: \"kubernetes.io/projected/8378fc38-80d3-4ca3-b413-c529bab97f09-kube-api-access-bwz82\") pod \"8378fc38-80d3-4ca3-b413-c529bab97f09\" (UID: \"8378fc38-80d3-4ca3-b413-c529bab97f09\") " Nov 03 21:00:24.935025 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:24.933329 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8378fc38-80d3-4ca3-b413-c529bab97f09-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8378fc38-80d3-4ca3-b413-c529bab97f09-maas-api-openshift-ca\") pod \"8378fc38-80d3-4ca3-b413-c529bab97f09\" (UID: \"8378fc38-80d3-4ca3-b413-c529bab97f09\") " Nov 03 21:00:24.935025 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:24.934757 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8378fc38-80d3-4ca3-b413-c529bab97f09-maas-api-openshift-ca" pod "8378fc38-80d3-4ca3-b413-c529bab97f09" (UID: "8378fc38-80d3-4ca3-b413-c529bab97f09"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:00:24.936799 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:24.936775 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac83ebbd-bxvgz" Nov 03 21:00:24.938253 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:24.938217 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8378fc38-80d3-4ca3-b413-c529bab97f09-maas-api-xks-ca" pod "8378fc38-80d3-4ca3-b413-c529bab97f09" (UID: "8378fc38-80d3-4ca3-b413-c529bab97f09"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:00:24.938994 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:24.938803 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8378fc38-80d3-4ca3-b413-c529bab97f09-kube-api-access-bwz82" pod "8378fc38-80d3-4ca3-b413-c529bab97f09" (UID: "8378fc38-80d3-4ca3-b413-c529bab97f09"). InnerVolumeSpecName "kube-api-access-bwz82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:24.939202 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:24.939184 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac83ebbd-bxvgz" event={"ID":"8378fc38-80d3-4ca3-b413-c529bab97f09","Type":"ContainerDied","Data":"efc187b58f1d47aab5770e42c7aada29b7a20dee1656d7a89a1e18673f347fe7"} Nov 03 21:00:24.939261 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:24.939234 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="efc187b58f1d47aab5770e42c7aada29b7a20dee1656d7a89a1e18673f347fe7" Nov 03 21:00:25.004726 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:25.004184 4305 patch_prober.go:29] interesting pod/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 03 21:00:25.004726 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:25.004255 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl" podUID="90f310bb-3fa1-404e-b893-6c49a2f760e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 03 21:00:25.046222 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:25.035569 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8378fc38-80d3-4ca3-b413-c529bab97f09-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:00:25.046222 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:25.035607 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwz82\" (UniqueName: \"kubernetes.io/projected/8378fc38-80d3-4ca3-b413-c529bab97f09-kube-api-access-bwz82\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:00:25.046222 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:25.035622 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8378fc38-80d3-4ca3-b413-c529bab97f09-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:00:25.978195 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:25.977786 4305 patch_prober.go:29] interesting pod/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 03 21:00:25.978195 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:25.977838 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl" podUID="90f310bb-3fa1-404e-b893-6c49a2f760e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 03 21:00:26.976902 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:26.976862 4305 patch_prober.go:29] interesting pod/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 03 21:00:26.977406 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:26.976925 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl" podUID="90f310bb-3fa1-404e-b893-6c49a2f760e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 03 21:00:27.977599 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:27.977550 4305 patch_prober.go:29] interesting pod/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 03 21:00:27.978144 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:27.977617 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl" podUID="90f310bb-3fa1-404e-b893-6c49a2f760e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 03 21:00:28.049618 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:28.049582 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c77973b3-ngzpp"] Nov 03 21:00:28.089282 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:28.089249 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c77973b3-ngzpp"] Nov 03 21:00:28.896888 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:28.896854 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8b10daa-ff51-4e8d-96e3-93e5c1740197" path="/var/lib/kubelet/pods/f8b10daa-ff51-4e8d-96e3-93e5c1740197/volumes" Nov 03 21:00:28.976631 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:28.976589 4305 patch_prober.go:29] interesting pod/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 03 21:00:28.976792 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:28.976672 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl" podUID="90f310bb-3fa1-404e-b893-6c49a2f760e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 03 21:00:29.974286 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:29.974253 4305 patch_prober.go:29] interesting pod/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 03 21:00:29.974692 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:29.974311 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl" podUID="90f310bb-3fa1-404e-b893-6c49a2f760e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 03 21:00:31.159797 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:31.159767 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl" Nov 03 21:00:31.160274 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:31.160177 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl" Nov 03 21:00:31.160995 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:31.160971 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl" Nov 03 21:00:36.038126 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.038092 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd"] Nov 03 21:00:36.038784 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.038769 4305 cpu_manager.go:414] "containerMap: removing container" podUID="18ce9b2e-c1e9-47e4-b29b-f2b272fef364" containerName="cleanup" Nov 03 21:00:36.038828 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.038785 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18ce9b2e-c1e9-47e4-b29b-f2b272fef364" containerName="cleanup" Nov 03 21:00:36.038828 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.038794 4305 cpu_manager.go:414] "containerMap: removing container" podUID="8378fc38-80d3-4ca3-b413-c529bab97f09" containerName="revoke-keys" Nov 03 21:00:36.038828 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.038799 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8378fc38-80d3-4ca3-b413-c529bab97f09" containerName="revoke-keys" Nov 03 21:00:36.038966 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.038890 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="18ce9b2e-c1e9-47e4-b29b-f2b272fef364" containerName="cleanup" Nov 03 21:00:36.038966 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.038898 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="8378fc38-80d3-4ca3-b413-c529bab97f09" containerName="revoke-keys" Nov 03 21:00:36.044110 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.044090 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd" Nov 03 21:00:36.113415 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.113378 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm84w\" (UniqueName: \"kubernetes.io/projected/94bcfeb9-714d-4f25-8ce7-c6c4577ea837-kube-api-access-fm84w\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd\" (UID: \"94bcfeb9-714d-4f25-8ce7-c6c4577ea837\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd" Nov 03 21:00:36.113645 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.113526 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94bcfeb9-714d-4f25-8ce7-c6c4577ea837-istiod-ca-cert\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd\" (UID: \"94bcfeb9-714d-4f25-8ce7-c6c4577ea837\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd" Nov 03 21:00:36.113645 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.113580 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94bcfeb9-714d-4f25-8ce7-c6c4577ea837-istio-podinfo\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd\" (UID: \"94bcfeb9-714d-4f25-8ce7-c6c4577ea837\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd" Nov 03 21:00:36.113751 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.113645 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94bcfeb9-714d-4f25-8ce7-c6c4577ea837-istio-data\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd\" (UID: \"94bcfeb9-714d-4f25-8ce7-c6c4577ea837\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd" Nov 03 21:00:36.113849 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.113821 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94bcfeb9-714d-4f25-8ce7-c6c4577ea837-workload-socket\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd\" (UID: \"94bcfeb9-714d-4f25-8ce7-c6c4577ea837\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd" Nov 03 21:00:36.113959 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.113942 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94bcfeb9-714d-4f25-8ce7-c6c4577ea837-istio-envoy\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd\" (UID: \"94bcfeb9-714d-4f25-8ce7-c6c4577ea837\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd" Nov 03 21:00:36.114024 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.114008 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94bcfeb9-714d-4f25-8ce7-c6c4577ea837-workload-certs\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd\" (UID: \"94bcfeb9-714d-4f25-8ce7-c6c4577ea837\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd" Nov 03 21:00:36.114058 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.114043 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94bcfeb9-714d-4f25-8ce7-c6c4577ea837-credential-socket\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd\" (UID: \"94bcfeb9-714d-4f25-8ce7-c6c4577ea837\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd" Nov 03 21:00:36.114098 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.114079 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94bcfeb9-714d-4f25-8ce7-c6c4577ea837-istio-token\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd\" (UID: \"94bcfeb9-714d-4f25-8ce7-c6c4577ea837\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd" Nov 03 21:00:36.148966 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.148929 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd"] Nov 03 21:00:36.215522 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.215484 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94bcfeb9-714d-4f25-8ce7-c6c4577ea837-istio-data\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd\" (UID: \"94bcfeb9-714d-4f25-8ce7-c6c4577ea837\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd" Nov 03 21:00:36.215742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.215588 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94bcfeb9-714d-4f25-8ce7-c6c4577ea837-workload-socket\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd\" (UID: \"94bcfeb9-714d-4f25-8ce7-c6c4577ea837\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd" Nov 03 21:00:36.215742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.215646 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94bcfeb9-714d-4f25-8ce7-c6c4577ea837-istio-envoy\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd\" (UID: \"94bcfeb9-714d-4f25-8ce7-c6c4577ea837\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd" Nov 03 21:00:36.215742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.215664 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94bcfeb9-714d-4f25-8ce7-c6c4577ea837-workload-certs\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd\" (UID: \"94bcfeb9-714d-4f25-8ce7-c6c4577ea837\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd" Nov 03 21:00:36.215742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.215688 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94bcfeb9-714d-4f25-8ce7-c6c4577ea837-credential-socket\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd\" (UID: \"94bcfeb9-714d-4f25-8ce7-c6c4577ea837\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd" Nov 03 21:00:36.215742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.215715 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94bcfeb9-714d-4f25-8ce7-c6c4577ea837-istio-token\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd\" (UID: \"94bcfeb9-714d-4f25-8ce7-c6c4577ea837\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd" Nov 03 21:00:36.216047 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.215772 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94bcfeb9-714d-4f25-8ce7-c6c4577ea837-istio-data\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd\" (UID: \"94bcfeb9-714d-4f25-8ce7-c6c4577ea837\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd" Nov 03 21:00:36.216047 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.215837 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm84w\" (UniqueName: \"kubernetes.io/projected/94bcfeb9-714d-4f25-8ce7-c6c4577ea837-kube-api-access-fm84w\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd\" (UID: \"94bcfeb9-714d-4f25-8ce7-c6c4577ea837\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd" Nov 03 21:00:36.216047 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.215904 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94bcfeb9-714d-4f25-8ce7-c6c4577ea837-istiod-ca-cert\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd\" (UID: \"94bcfeb9-714d-4f25-8ce7-c6c4577ea837\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd" Nov 03 21:00:36.216047 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.215974 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94bcfeb9-714d-4f25-8ce7-c6c4577ea837-istio-podinfo\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd\" (UID: \"94bcfeb9-714d-4f25-8ce7-c6c4577ea837\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd" Nov 03 21:00:36.216047 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.215985 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94bcfeb9-714d-4f25-8ce7-c6c4577ea837-workload-socket\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd\" (UID: \"94bcfeb9-714d-4f25-8ce7-c6c4577ea837\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd" Nov 03 21:00:36.216047 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.216011 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94bcfeb9-714d-4f25-8ce7-c6c4577ea837-workload-certs\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd\" (UID: \"94bcfeb9-714d-4f25-8ce7-c6c4577ea837\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd" Nov 03 21:00:36.216285 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.216065 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94bcfeb9-714d-4f25-8ce7-c6c4577ea837-credential-socket\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd\" (UID: \"94bcfeb9-714d-4f25-8ce7-c6c4577ea837\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd" Nov 03 21:00:36.216617 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.216599 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94bcfeb9-714d-4f25-8ce7-c6c4577ea837-istiod-ca-cert\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd\" (UID: \"94bcfeb9-714d-4f25-8ce7-c6c4577ea837\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd" Nov 03 21:00:36.218133 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.218112 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94bcfeb9-714d-4f25-8ce7-c6c4577ea837-istio-envoy\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd\" (UID: \"94bcfeb9-714d-4f25-8ce7-c6c4577ea837\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd" Nov 03 21:00:36.218383 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.218367 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94bcfeb9-714d-4f25-8ce7-c6c4577ea837-istio-podinfo\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd\" (UID: \"94bcfeb9-714d-4f25-8ce7-c6c4577ea837\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd" Nov 03 21:00:36.276910 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.276876 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94bcfeb9-714d-4f25-8ce7-c6c4577ea837-istio-token\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd\" (UID: \"94bcfeb9-714d-4f25-8ce7-c6c4577ea837\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd" Nov 03 21:00:36.277096 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.276882 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm84w\" (UniqueName: \"kubernetes.io/projected/94bcfeb9-714d-4f25-8ce7-c6c4577ea837-kube-api-access-fm84w\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd\" (UID: \"94bcfeb9-714d-4f25-8ce7-c6c4577ea837\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd" Nov 03 21:00:36.370182 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.370100 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd" Nov 03 21:00:36.398146 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.398114 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd"] Nov 03 21:00:36.403173 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.403147 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd" Nov 03 21:00:36.420400 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.420366 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d"] Nov 03 21:00:36.426129 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.426109 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d" Nov 03 21:00:36.458844 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.458758 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd"] Nov 03 21:00:36.484599 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.484569 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d"] Nov 03 21:00:36.520264 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.520120 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ff154f8-a2c6-49a4-bbc8-f5026ec757bd-workload-certs\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd\" (UID: \"0ff154f8-a2c6-49a4-bbc8-f5026ec757bd\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd" Nov 03 21:00:36.520264 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.520178 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9103e89-52f2-4022-bd9b-dc554c82f479-istio-data\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d\" (UID: \"d9103e89-52f2-4022-bd9b-dc554c82f479\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d" Nov 03 21:00:36.520264 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.520225 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9103e89-52f2-4022-bd9b-dc554c82f479-istio-token\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d\" (UID: \"d9103e89-52f2-4022-bd9b-dc554c82f479\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d" Nov 03 21:00:36.520264 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.520247 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqzwj\" (UniqueName: \"kubernetes.io/projected/d9103e89-52f2-4022-bd9b-dc554c82f479-kube-api-access-lqzwj\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d\" (UID: \"d9103e89-52f2-4022-bd9b-dc554c82f479\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d" Nov 03 21:00:36.520264 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.520266 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ff154f8-a2c6-49a4-bbc8-f5026ec757bd-istio-token\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd\" (UID: \"0ff154f8-a2c6-49a4-bbc8-f5026ec757bd\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd" Nov 03 21:00:36.521085 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.521049 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ff154f8-a2c6-49a4-bbc8-f5026ec757bd-istiod-ca-cert\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd\" (UID: \"0ff154f8-a2c6-49a4-bbc8-f5026ec757bd\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd" Nov 03 21:00:36.521185 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.521125 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9103e89-52f2-4022-bd9b-dc554c82f479-workload-certs\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d\" (UID: \"d9103e89-52f2-4022-bd9b-dc554c82f479\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d" Nov 03 21:00:36.521185 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.521172 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9103e89-52f2-4022-bd9b-dc554c82f479-istio-envoy\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d\" (UID: \"d9103e89-52f2-4022-bd9b-dc554c82f479\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d" Nov 03 21:00:36.521292 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.521204 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9103e89-52f2-4022-bd9b-dc554c82f479-workload-socket\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d\" (UID: \"d9103e89-52f2-4022-bd9b-dc554c82f479\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d" Nov 03 21:00:36.521806 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.521785 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9103e89-52f2-4022-bd9b-dc554c82f479-istio-podinfo\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d\" (UID: \"d9103e89-52f2-4022-bd9b-dc554c82f479\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d" Nov 03 21:00:36.521887 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.521840 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff154f8-a2c6-49a4-bbc8-f5026ec757bd-workload-socket\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd\" (UID: \"0ff154f8-a2c6-49a4-bbc8-f5026ec757bd\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd" Nov 03 21:00:36.522067 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.522049 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff154f8-a2c6-49a4-bbc8-f5026ec757bd-credential-socket\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd\" (UID: \"0ff154f8-a2c6-49a4-bbc8-f5026ec757bd\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd" Nov 03 21:00:36.522117 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.522083 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ff154f8-a2c6-49a4-bbc8-f5026ec757bd-istio-envoy\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd\" (UID: \"0ff154f8-a2c6-49a4-bbc8-f5026ec757bd\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd" Nov 03 21:00:36.522165 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.522133 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9103e89-52f2-4022-bd9b-dc554c82f479-credential-socket\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d\" (UID: \"d9103e89-52f2-4022-bd9b-dc554c82f479\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d" Nov 03 21:00:36.522215 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.522163 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ff154f8-a2c6-49a4-bbc8-f5026ec757bd-istio-podinfo\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd\" (UID: \"0ff154f8-a2c6-49a4-bbc8-f5026ec757bd\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd" Nov 03 21:00:36.522265 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.522215 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ff154f8-a2c6-49a4-bbc8-f5026ec757bd-istio-data\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd\" (UID: \"0ff154f8-a2c6-49a4-bbc8-f5026ec757bd\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd" Nov 03 21:00:36.522265 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.522256 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9103e89-52f2-4022-bd9b-dc554c82f479-istiod-ca-cert\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d\" (UID: \"d9103e89-52f2-4022-bd9b-dc554c82f479\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d" Nov 03 21:00:36.522358 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.522334 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tkls\" (UniqueName: \"kubernetes.io/projected/0ff154f8-a2c6-49a4-bbc8-f5026ec757bd-kube-api-access-6tkls\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd\" (UID: \"0ff154f8-a2c6-49a4-bbc8-f5026ec757bd\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd" Nov 03 21:00:36.623989 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.623873 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9103e89-52f2-4022-bd9b-dc554c82f479-workload-certs\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d\" (UID: \"d9103e89-52f2-4022-bd9b-dc554c82f479\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d" Nov 03 21:00:36.623989 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.623930 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9103e89-52f2-4022-bd9b-dc554c82f479-istio-envoy\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d\" (UID: \"d9103e89-52f2-4022-bd9b-dc554c82f479\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d" Nov 03 21:00:36.623989 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.623972 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9103e89-52f2-4022-bd9b-dc554c82f479-workload-socket\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d\" (UID: \"d9103e89-52f2-4022-bd9b-dc554c82f479\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d" Nov 03 21:00:36.624279 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.624000 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9103e89-52f2-4022-bd9b-dc554c82f479-istio-podinfo\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d\" (UID: \"d9103e89-52f2-4022-bd9b-dc554c82f479\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d" Nov 03 21:00:36.624279 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.624023 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff154f8-a2c6-49a4-bbc8-f5026ec757bd-workload-socket\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd\" (UID: \"0ff154f8-a2c6-49a4-bbc8-f5026ec757bd\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd" Nov 03 21:00:36.624279 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.624069 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff154f8-a2c6-49a4-bbc8-f5026ec757bd-credential-socket\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd\" (UID: \"0ff154f8-a2c6-49a4-bbc8-f5026ec757bd\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd" Nov 03 21:00:36.624279 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.624197 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ff154f8-a2c6-49a4-bbc8-f5026ec757bd-istio-envoy\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd\" (UID: \"0ff154f8-a2c6-49a4-bbc8-f5026ec757bd\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd" Nov 03 21:00:36.624279 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.624266 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9103e89-52f2-4022-bd9b-dc554c82f479-credential-socket\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d\" (UID: \"d9103e89-52f2-4022-bd9b-dc554c82f479\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d" Nov 03 21:00:36.624565 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.624299 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ff154f8-a2c6-49a4-bbc8-f5026ec757bd-istio-podinfo\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd\" (UID: \"0ff154f8-a2c6-49a4-bbc8-f5026ec757bd\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd" Nov 03 21:00:36.624565 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.624353 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9103e89-52f2-4022-bd9b-dc554c82f479-workload-certs\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d\" (UID: \"d9103e89-52f2-4022-bd9b-dc554c82f479\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d" Nov 03 21:00:36.624565 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.624366 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ff154f8-a2c6-49a4-bbc8-f5026ec757bd-istio-data\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd\" (UID: \"0ff154f8-a2c6-49a4-bbc8-f5026ec757bd\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd" Nov 03 21:00:36.624565 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.624417 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9103e89-52f2-4022-bd9b-dc554c82f479-istiod-ca-cert\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d\" (UID: \"d9103e89-52f2-4022-bd9b-dc554c82f479\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d" Nov 03 21:00:36.624565 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.624468 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff154f8-a2c6-49a4-bbc8-f5026ec757bd-workload-socket\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd\" (UID: \"0ff154f8-a2c6-49a4-bbc8-f5026ec757bd\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd" Nov 03 21:00:36.624565 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.624487 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9103e89-52f2-4022-bd9b-dc554c82f479-workload-socket\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d\" (UID: \"d9103e89-52f2-4022-bd9b-dc554c82f479\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d" Nov 03 21:00:36.624565 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.624555 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tkls\" (UniqueName: \"kubernetes.io/projected/0ff154f8-a2c6-49a4-bbc8-f5026ec757bd-kube-api-access-6tkls\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd\" (UID: \"0ff154f8-a2c6-49a4-bbc8-f5026ec757bd\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd" Nov 03 21:00:36.625093 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.624617 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ff154f8-a2c6-49a4-bbc8-f5026ec757bd-workload-certs\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd\" (UID: \"0ff154f8-a2c6-49a4-bbc8-f5026ec757bd\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd" Nov 03 21:00:36.625093 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.624665 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9103e89-52f2-4022-bd9b-dc554c82f479-istio-data\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d\" (UID: \"d9103e89-52f2-4022-bd9b-dc554c82f479\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d" Nov 03 21:00:36.625093 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.624786 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ff154f8-a2c6-49a4-bbc8-f5026ec757bd-istio-data\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd\" (UID: \"0ff154f8-a2c6-49a4-bbc8-f5026ec757bd\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd" Nov 03 21:00:36.625093 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.624885 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9103e89-52f2-4022-bd9b-dc554c82f479-istio-token\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d\" (UID: \"d9103e89-52f2-4022-bd9b-dc554c82f479\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d" Nov 03 21:00:36.625093 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.624919 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqzwj\" (UniqueName: \"kubernetes.io/projected/d9103e89-52f2-4022-bd9b-dc554c82f479-kube-api-access-lqzwj\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d\" (UID: \"d9103e89-52f2-4022-bd9b-dc554c82f479\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d" Nov 03 21:00:36.625093 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.624948 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ff154f8-a2c6-49a4-bbc8-f5026ec757bd-istio-token\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd\" (UID: \"0ff154f8-a2c6-49a4-bbc8-f5026ec757bd\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd" Nov 03 21:00:36.625093 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.625088 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9103e89-52f2-4022-bd9b-dc554c82f479-istiod-ca-cert\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d\" (UID: \"d9103e89-52f2-4022-bd9b-dc554c82f479\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d" Nov 03 21:00:36.625467 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.624494 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff154f8-a2c6-49a4-bbc8-f5026ec757bd-credential-socket\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd\" (UID: \"0ff154f8-a2c6-49a4-bbc8-f5026ec757bd\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd" Nov 03 21:00:36.625467 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.625350 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ff154f8-a2c6-49a4-bbc8-f5026ec757bd-workload-certs\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd\" (UID: \"0ff154f8-a2c6-49a4-bbc8-f5026ec757bd\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd" Nov 03 21:00:36.625467 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.625424 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ff154f8-a2c6-49a4-bbc8-f5026ec757bd-istiod-ca-cert\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd\" (UID: \"0ff154f8-a2c6-49a4-bbc8-f5026ec757bd\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd" Nov 03 21:00:36.625629 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.625568 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9103e89-52f2-4022-bd9b-dc554c82f479-istio-data\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d\" (UID: \"d9103e89-52f2-4022-bd9b-dc554c82f479\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d" Nov 03 21:00:36.625707 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.625686 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9103e89-52f2-4022-bd9b-dc554c82f479-credential-socket\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d\" (UID: \"d9103e89-52f2-4022-bd9b-dc554c82f479\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d" Nov 03 21:00:36.626231 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.626206 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ff154f8-a2c6-49a4-bbc8-f5026ec757bd-istiod-ca-cert\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd\" (UID: \"0ff154f8-a2c6-49a4-bbc8-f5026ec757bd\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd" Nov 03 21:00:36.627047 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.627023 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9103e89-52f2-4022-bd9b-dc554c82f479-istio-podinfo\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d\" (UID: \"d9103e89-52f2-4022-bd9b-dc554c82f479\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d" Nov 03 21:00:36.627203 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.627182 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9103e89-52f2-4022-bd9b-dc554c82f479-istio-envoy\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d\" (UID: \"d9103e89-52f2-4022-bd9b-dc554c82f479\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d" Nov 03 21:00:36.627274 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.627202 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ff154f8-a2c6-49a4-bbc8-f5026ec757bd-istio-envoy\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd\" (UID: \"0ff154f8-a2c6-49a4-bbc8-f5026ec757bd\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd" Nov 03 21:00:36.627826 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.627801 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ff154f8-a2c6-49a4-bbc8-f5026ec757bd-istio-podinfo\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd\" (UID: \"0ff154f8-a2c6-49a4-bbc8-f5026ec757bd\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd" Nov 03 21:00:36.688776 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.688737 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ff154f8-a2c6-49a4-bbc8-f5026ec757bd-istio-token\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd\" (UID: \"0ff154f8-a2c6-49a4-bbc8-f5026ec757bd\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd" Nov 03 21:00:36.688776 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.688741 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9103e89-52f2-4022-bd9b-dc554c82f479-istio-token\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d\" (UID: \"d9103e89-52f2-4022-bd9b-dc554c82f479\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d" Nov 03 21:00:36.689006 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.688898 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tkls\" (UniqueName: \"kubernetes.io/projected/0ff154f8-a2c6-49a4-bbc8-f5026ec757bd-kube-api-access-6tkls\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd\" (UID: \"0ff154f8-a2c6-49a4-bbc8-f5026ec757bd\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd" Nov 03 21:00:36.693564 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.693539 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqzwj\" (UniqueName: \"kubernetes.io/projected/d9103e89-52f2-4022-bd9b-dc554c82f479-kube-api-access-lqzwj\") pod \"e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d\" (UID: \"d9103e89-52f2-4022-bd9b-dc554c82f479\") " pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d" Nov 03 21:00:36.743732 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.743705 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd" Nov 03 21:00:36.769789 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.769754 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d" Nov 03 21:00:36.853794 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:00:36.853768 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94bcfeb9_714d_4f25_8ce7_c6c4577ea837.slice/crio-c19d07edeaf6cf3be9b0a38d9ab7456d5e5888983f1a3712393d4063195b3947 WatchSource:0}: Error finding container c19d07edeaf6cf3be9b0a38d9ab7456d5e5888983f1a3712393d4063195b3947: Status 404 returned error can't find the container with id c19d07edeaf6cf3be9b0a38d9ab7456d5e5888983f1a3712393d4063195b3947 Nov 03 21:00:36.854675 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.854653 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd"] Nov 03 21:00:36.856474 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.856421 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:00:36.856579 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.856520 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:00:36.856579 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:36.856566 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:00:37.052202 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:37.052167 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a58fa67a-s5r5h"] Nov 03 21:00:37.056620 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:37.056592 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a58fa67a-s5r5h"] Nov 03 21:00:37.067404 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:37.067372 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd" event={"ID":"94bcfeb9-714d-4f25-8ce7-c6c4577ea837","Type":"ContainerStarted","Data":"b1008eb64b520f663b7e82c72eecf11ea5d8037cbd6c6f312cfe85e5a20e97b7"} Nov 03 21:00:37.067601 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:37.067449 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd" event={"ID":"94bcfeb9-714d-4f25-8ce7-c6c4577ea837","Type":"ContainerStarted","Data":"c19d07edeaf6cf3be9b0a38d9ab7456d5e5888983f1a3712393d4063195b3947"} Nov 03 21:00:37.130720 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:00:37.130693 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ff154f8_a2c6_49a4_bbc8_f5026ec757bd.slice/crio-57c630e392d5609a81722ba5ba748f66d488fcc6996229f9b33f31bceef7eac2 WatchSource:0}: Error finding container 57c630e392d5609a81722ba5ba748f66d488fcc6996229f9b33f31bceef7eac2: Status 404 returned error can't find the container with id 57c630e392d5609a81722ba5ba748f66d488fcc6996229f9b33f31bceef7eac2 Nov 03 21:00:37.130966 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:37.130947 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd"] Nov 03 21:00:37.133645 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:37.133609 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:00:37.133743 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:37.133688 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:00:37.133743 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:37.133735 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:00:37.180202 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:00:37.180168 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9103e89_52f2_4022_bd9b_dc554c82f479.slice/crio-8090be0882cf15822b6704f55a8398f02bcdea74e215a356c3361cfdc39e69f7 WatchSource:0}: Error finding container 8090be0882cf15822b6704f55a8398f02bcdea74e215a356c3361cfdc39e69f7: Status 404 returned error can't find the container with id 8090be0882cf15822b6704f55a8398f02bcdea74e215a356c3361cfdc39e69f7 Nov 03 21:00:37.182839 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:37.182804 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d"] Nov 03 21:00:37.183065 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:37.183034 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:00:37.183138 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:37.183099 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:00:37.183138 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:37.183127 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:00:37.196010 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:37.194903 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd" podStartSLOduration=2.194881189 podStartE2EDuration="2.194881189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:37.192948294 +0000 UTC m=+1272.418268109" watchObservedRunningTime="2026-11-03 21:00:37.194881189 +0000 UTC m=+1272.420200981" Nov 03 21:00:37.371103 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:37.371069 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd" Nov 03 21:00:37.372562 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:37.372536 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 03 21:00:37.372692 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:37.372585 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd" podUID="94bcfeb9-714d-4f25-8ce7-c6c4577ea837" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 03 21:00:38.107774 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:38.107241 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d" event={"ID":"d9103e89-52f2-4022-bd9b-dc554c82f479","Type":"ContainerStarted","Data":"d950b9777f3fe72d899dfe98f108d37bb059466e0e8b4c1b5216870ecd4bafcc"} Nov 03 21:00:38.107774 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:38.107312 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d" event={"ID":"d9103e89-52f2-4022-bd9b-dc554c82f479","Type":"ContainerStarted","Data":"8090be0882cf15822b6704f55a8398f02bcdea74e215a356c3361cfdc39e69f7"} Nov 03 21:00:38.112825 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:38.112658 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd" event={"ID":"0ff154f8-a2c6-49a4-bbc8-f5026ec757bd","Type":"ContainerStarted","Data":"4d111a68c0e46f09480dfdfe969cb1010022c19b4dd169cff64e9ee661f2334c"} Nov 03 21:00:38.112825 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:38.112715 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd" event={"ID":"0ff154f8-a2c6-49a4-bbc8-f5026ec757bd","Type":"ContainerStarted","Data":"57c630e392d5609a81722ba5ba748f66d488fcc6996229f9b33f31bceef7eac2"} Nov 03 21:00:38.245500 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:38.245423 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d" podStartSLOduration=2.245406381 podStartE2EDuration="2.245406381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:38.228149077 +0000 UTC m=+1273.453468870" watchObservedRunningTime="2026-11-03 21:00:38.245406381 +0000 UTC m=+1273.470726170" Nov 03 21:00:38.375295 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:38.375196 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 03 21:00:38.376994 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:38.376953 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd" podUID="94bcfeb9-714d-4f25-8ce7-c6c4577ea837" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 03 21:00:38.393804 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:38.393744 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd" podStartSLOduration=2.393723516 podStartE2EDuration="2.393723516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:38.374808062 +0000 UTC m=+1273.600127879" watchObservedRunningTime="2026-11-03 21:00:38.393723516 +0000 UTC m=+1273.619043307" Nov 03 21:00:38.744507 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:38.744469 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd" Nov 03 21:00:38.745952 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:38.745930 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 03 21:00:38.746023 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:38.745981 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd" podUID="0ff154f8-a2c6-49a4-bbc8-f5026ec757bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 03 21:00:38.771460 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:38.770813 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d" Nov 03 21:00:38.781473 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:38.780572 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 21:00:38.781473 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:38.780657 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d" podUID="d9103e89-52f2-4022-bd9b-dc554c82f479" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 21:00:38.940183 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:38.939940 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6b6b70b-9fc3-4786-bf43-afedba0e31a3" path="/var/lib/kubelet/pods/b6b6b70b-9fc3-4786-bf43-afedba0e31a3/volumes" Nov 03 21:00:38.940675 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:38.940647 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13e3f48a-ql8qx"] Nov 03 21:00:38.946178 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:38.946155 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e3f48a-ql8qx" Nov 03 21:00:39.038056 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:39.011271 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:00:39.038056 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:39.033265 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77vtd\"" Nov 03 21:00:39.038056 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:39.033352 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:00:39.038056 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:39.033407 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:00:39.038056 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:39.035338 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12b59755-1aa9-4448-b884-cc45ad03be96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13e3f48a-ql8qx\" (UID: \"12b59755-1aa9-4448-b884-cc45ad03be96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e3f48a-ql8qx" Nov 03 21:00:39.038056 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:39.035393 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12b59755-1aa9-4448-b884-cc45ad03be96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13e3f48a-ql8qx\" (UID: \"12b59755-1aa9-4448-b884-cc45ad03be96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e3f48a-ql8qx" Nov 03 21:00:39.038056 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:39.035506 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6lpg\" (UniqueName: \"kubernetes.io/projected/12b59755-1aa9-4448-b884-cc45ad03be96-kube-api-access-d6lpg\") pod \"maas-api-revoke-sub-13e3f48a-ql8qx\" (UID: \"12b59755-1aa9-4448-b884-cc45ad03be96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e3f48a-ql8qx" Nov 03 21:00:39.052459 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:39.050674 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13e3f48a-ql8qx"] Nov 03 21:00:39.141457 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:39.139982 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2f65010-2rf7k"] Nov 03 21:00:39.141848 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:39.141754 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2f65010-2rf7k"] Nov 03 21:00:39.147267 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:39.145465 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12b59755-1aa9-4448-b884-cc45ad03be96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13e3f48a-ql8qx\" (UID: \"12b59755-1aa9-4448-b884-cc45ad03be96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e3f48a-ql8qx" Nov 03 21:00:39.147267 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:39.145530 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12b59755-1aa9-4448-b884-cc45ad03be96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13e3f48a-ql8qx\" (UID: \"12b59755-1aa9-4448-b884-cc45ad03be96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e3f48a-ql8qx" Nov 03 21:00:39.147267 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:39.145623 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6lpg\" (UniqueName: \"kubernetes.io/projected/12b59755-1aa9-4448-b884-cc45ad03be96-kube-api-access-d6lpg\") pod \"maas-api-revoke-sub-13e3f48a-ql8qx\" (UID: \"12b59755-1aa9-4448-b884-cc45ad03be96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e3f48a-ql8qx" Nov 03 21:00:39.147267 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:39.146868 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12b59755-1aa9-4448-b884-cc45ad03be96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13e3f48a-ql8qx\" (UID: \"12b59755-1aa9-4448-b884-cc45ad03be96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e3f48a-ql8qx" Nov 03 21:00:39.149959 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:39.149934 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12b59755-1aa9-4448-b884-cc45ad03be96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13e3f48a-ql8qx\" (UID: \"12b59755-1aa9-4448-b884-cc45ad03be96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e3f48a-ql8qx" Nov 03 21:00:39.240618 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:39.240579 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6lpg\" (UniqueName: \"kubernetes.io/projected/12b59755-1aa9-4448-b884-cc45ad03be96-kube-api-access-d6lpg\") pod \"maas-api-revoke-sub-13e3f48a-ql8qx\" (UID: \"12b59755-1aa9-4448-b884-cc45ad03be96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e3f48a-ql8qx" Nov 03 21:00:39.338638 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:39.338557 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e3f48a-ql8qx" Nov 03 21:00:39.374309 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:39.373845 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 03 21:00:39.374309 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:39.373924 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd" podUID="94bcfeb9-714d-4f25-8ce7-c6c4577ea837" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 03 21:00:39.749338 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:39.749309 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 03 21:00:39.749557 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:39.749525 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd" podUID="0ff154f8-a2c6-49a4-bbc8-f5026ec757bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 03 21:00:39.785457 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:39.785213 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 21:00:39.785457 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:39.785277 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d" podUID="d9103e89-52f2-4022-bd9b-dc554c82f479" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 21:00:40.137511 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:40.137482 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13e3f48a-ql8qx"] Nov 03 21:00:40.139698 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:00:40.139660 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12b59755_1aa9_4448_b884_cc45ad03be96.slice/crio-fae0e2c60e9ed480fb40c2c711132f04a16052ec5c81092eb82a56010a2fc56d WatchSource:0}: Error finding container fae0e2c60e9ed480fb40c2c711132f04a16052ec5c81092eb82a56010a2fc56d: Status 404 returned error can't find the container with id fae0e2c60e9ed480fb40c2c711132f04a16052ec5c81092eb82a56010a2fc56d Nov 03 21:00:40.202473 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:40.183670 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e3f48a-ql8qx" event={"ID":"12b59755-1aa9-4448-b884-cc45ad03be96","Type":"ContainerStarted","Data":"fae0e2c60e9ed480fb40c2c711132f04a16052ec5c81092eb82a56010a2fc56d"} Nov 03 21:00:40.387520 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:40.373851 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 03 21:00:40.387520 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:40.373915 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd" podUID="94bcfeb9-714d-4f25-8ce7-c6c4577ea837" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 03 21:00:40.748453 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:40.745469 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 03 21:00:40.748453 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:40.745523 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd" podUID="0ff154f8-a2c6-49a4-bbc8-f5026ec757bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 03 21:00:40.779455 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:40.779171 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 21:00:40.779455 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:40.779246 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d" podUID="d9103e89-52f2-4022-bd9b-dc554c82f479" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 21:00:40.972371 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:40.972326 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f7a55ca-e2b1-469e-b5fa-22d4558de0fb" path="/var/lib/kubelet/pods/4f7a55ca-e2b1-469e-b5fa-22d4558de0fb/volumes" Nov 03 21:00:41.343782 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:41.343741 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e3f48a-ql8qx" event={"ID":"12b59755-1aa9-4448-b884-cc45ad03be96","Type":"ContainerStarted","Data":"982701db3b1034b59f2c8924e4608dfa84a079f8e999739fe329f0ec42ea2b27"} Nov 03 21:00:41.384024 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:41.383986 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 03 21:00:41.384326 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:41.384298 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd" podUID="94bcfeb9-714d-4f25-8ce7-c6c4577ea837" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 03 21:00:41.550627 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:41.550571 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e3f48a-ql8qx" podStartSLOduration=3.550548027 podStartE2EDuration="3.550548027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:41.494619833 +0000 UTC m=+1276.719939624" watchObservedRunningTime="2026-11-03 21:00:41.550548027 +0000 UTC m=+1276.775867817" Nov 03 21:00:41.745138 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:41.745108 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 03 21:00:41.745321 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:41.745153 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd" podUID="0ff154f8-a2c6-49a4-bbc8-f5026ec757bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 03 21:00:41.802462 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:41.798927 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 21:00:41.802462 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:41.799003 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d" podUID="d9103e89-52f2-4022-bd9b-dc554c82f479" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 21:00:42.377459 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:42.376627 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 03 21:00:42.377459 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:42.376694 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd" podUID="94bcfeb9-714d-4f25-8ce7-c6c4577ea837" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 03 21:00:42.434857 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:42.434807 4305 generic.go:356] "Generic (PLEG): container finished" podID="12b59755-1aa9-4448-b884-cc45ad03be96" containerID="982701db3b1034b59f2c8924e4608dfa84a079f8e999739fe329f0ec42ea2b27" exitCode=0 Nov 03 21:00:42.435049 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:42.434891 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e3f48a-ql8qx" event={"ID":"12b59755-1aa9-4448-b884-cc45ad03be96","Type":"ContainerDied","Data":"982701db3b1034b59f2c8924e4608dfa84a079f8e999739fe329f0ec42ea2b27"} Nov 03 21:00:42.744802 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:42.744760 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 03 21:00:42.744992 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:42.744825 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd" podUID="0ff154f8-a2c6-49a4-bbc8-f5026ec757bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 03 21:00:42.826466 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:42.825606 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 21:00:42.826466 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:42.825688 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d" podUID="d9103e89-52f2-4022-bd9b-dc554c82f479" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 21:00:43.146459 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:43.146178 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c034f31-xqldb"] Nov 03 21:00:43.150627 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:43.150607 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c034f31-xqldb" Nov 03 21:00:43.198585 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:43.198540 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6045b8c9-a516-4e66-8421-62fe0d7fbc79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c034f31-xqldb\" (UID: \"6045b8c9-a516-4e66-8421-62fe0d7fbc79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c034f31-xqldb" Nov 03 21:00:43.198838 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:43.198820 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7nwk\" (UniqueName: \"kubernetes.io/projected/6045b8c9-a516-4e66-8421-62fe0d7fbc79-kube-api-access-r7nwk\") pod \"maas-api-revoke-sub-3c034f31-xqldb\" (UID: \"6045b8c9-a516-4e66-8421-62fe0d7fbc79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c034f31-xqldb" Nov 03 21:00:43.199162 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:43.199144 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6045b8c9-a516-4e66-8421-62fe0d7fbc79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c034f31-xqldb\" (UID: \"6045b8c9-a516-4e66-8421-62fe0d7fbc79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c034f31-xqldb" Nov 03 21:00:43.218627 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:43.218588 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c034f31-xqldb"] Nov 03 21:00:43.310180 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:43.310138 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6045b8c9-a516-4e66-8421-62fe0d7fbc79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c034f31-xqldb\" (UID: \"6045b8c9-a516-4e66-8421-62fe0d7fbc79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c034f31-xqldb" Nov 03 21:00:43.310589 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:43.310566 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7nwk\" (UniqueName: \"kubernetes.io/projected/6045b8c9-a516-4e66-8421-62fe0d7fbc79-kube-api-access-r7nwk\") pod \"maas-api-revoke-sub-3c034f31-xqldb\" (UID: \"6045b8c9-a516-4e66-8421-62fe0d7fbc79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c034f31-xqldb" Nov 03 21:00:43.310970 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:43.310951 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6045b8c9-a516-4e66-8421-62fe0d7fbc79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c034f31-xqldb\" (UID: \"6045b8c9-a516-4e66-8421-62fe0d7fbc79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c034f31-xqldb" Nov 03 21:00:43.314924 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:43.314172 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6045b8c9-a516-4e66-8421-62fe0d7fbc79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c034f31-xqldb\" (UID: \"6045b8c9-a516-4e66-8421-62fe0d7fbc79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c034f31-xqldb" Nov 03 21:00:43.316914 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:43.316881 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6045b8c9-a516-4e66-8421-62fe0d7fbc79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c034f31-xqldb\" (UID: \"6045b8c9-a516-4e66-8421-62fe0d7fbc79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c034f31-xqldb" Nov 03 21:00:43.372461 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:43.370637 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 03 21:00:43.372461 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:43.370714 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd" podUID="94bcfeb9-714d-4f25-8ce7-c6c4577ea837" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 03 21:00:43.401507 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:43.400353 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7nwk\" (UniqueName: \"kubernetes.io/projected/6045b8c9-a516-4e66-8421-62fe0d7fbc79-kube-api-access-r7nwk\") pod \"maas-api-revoke-sub-3c034f31-xqldb\" (UID: \"6045b8c9-a516-4e66-8421-62fe0d7fbc79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c034f31-xqldb" Nov 03 21:00:43.520004 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:43.519973 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c034f31-xqldb" Nov 03 21:00:43.731541 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:43.729657 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e3f48a-ql8qx" Nov 03 21:00:43.744751 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:43.744721 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 03 21:00:43.744892 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:43.744776 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd" podUID="0ff154f8-a2c6-49a4-bbc8-f5026ec757bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 03 21:00:43.770734 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:43.770700 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 21:00:43.770893 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:43.770755 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d" podUID="d9103e89-52f2-4022-bd9b-dc554c82f479" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 21:00:43.821595 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:43.821552 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12b59755-1aa9-4448-b884-cc45ad03be96-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12b59755-1aa9-4448-b884-cc45ad03be96-maas-api-openshift-ca\") pod \"12b59755-1aa9-4448-b884-cc45ad03be96\" (UID: \"12b59755-1aa9-4448-b884-cc45ad03be96\") " Nov 03 21:00:43.821775 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:43.821614 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12b59755-1aa9-4448-b884-cc45ad03be96-kube-api-access-d6lpg\" (UniqueName: \"kubernetes.io/projected/12b59755-1aa9-4448-b884-cc45ad03be96-kube-api-access-d6lpg\") pod \"12b59755-1aa9-4448-b884-cc45ad03be96\" (UID: \"12b59755-1aa9-4448-b884-cc45ad03be96\") " Nov 03 21:00:43.821775 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:43.821728 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12b59755-1aa9-4448-b884-cc45ad03be96-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12b59755-1aa9-4448-b884-cc45ad03be96-maas-api-xks-ca\") pod \"12b59755-1aa9-4448-b884-cc45ad03be96\" (UID: \"12b59755-1aa9-4448-b884-cc45ad03be96\") " Nov 03 21:00:43.822044 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:43.822002 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12b59755-1aa9-4448-b884-cc45ad03be96-maas-api-openshift-ca" pod "12b59755-1aa9-4448-b884-cc45ad03be96" (UID: "12b59755-1aa9-4448-b884-cc45ad03be96"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:00:43.822514 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:43.822473 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12b59755-1aa9-4448-b884-cc45ad03be96-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:00:43.823983 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:43.823943 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12b59755-1aa9-4448-b884-cc45ad03be96-maas-api-xks-ca" pod "12b59755-1aa9-4448-b884-cc45ad03be96" (UID: "12b59755-1aa9-4448-b884-cc45ad03be96"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:00:43.824504 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:43.824477 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12b59755-1aa9-4448-b884-cc45ad03be96-kube-api-access-d6lpg" pod "12b59755-1aa9-4448-b884-cc45ad03be96" (UID: "12b59755-1aa9-4448-b884-cc45ad03be96"). InnerVolumeSpecName "kube-api-access-d6lpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:43.925514 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:43.924708 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6lpg\" (UniqueName: \"kubernetes.io/projected/12b59755-1aa9-4448-b884-cc45ad03be96-kube-api-access-d6lpg\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:00:43.925514 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:43.924743 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12b59755-1aa9-4448-b884-cc45ad03be96-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:00:43.988084 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:43.988052 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c034f31-xqldb"] Nov 03 21:00:43.989129 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:00:43.989102 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6045b8c9_a516_4e66_8421_62fe0d7fbc79.slice/crio-e821330cab246528cea2c19883521e104f907ef261ee4b60da0aaf64dcd46d8c WatchSource:0}: Error finding container e821330cab246528cea2c19883521e104f907ef261ee4b60da0aaf64dcd46d8c: Status 404 returned error can't find the container with id e821330cab246528cea2c19883521e104f907ef261ee4b60da0aaf64dcd46d8c Nov 03 21:00:44.371298 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:44.371256 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 03 21:00:44.371515 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:44.371317 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd" podUID="94bcfeb9-714d-4f25-8ce7-c6c4577ea837" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 03 21:00:44.487455 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:44.487126 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e3f48a-ql8qx" event={"ID":"12b59755-1aa9-4448-b884-cc45ad03be96","Type":"ContainerDied","Data":"fae0e2c60e9ed480fb40c2c711132f04a16052ec5c81092eb82a56010a2fc56d"} Nov 03 21:00:44.488695 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:44.487188 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fae0e2c60e9ed480fb40c2c711132f04a16052ec5c81092eb82a56010a2fc56d" Nov 03 21:00:44.488695 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:44.487264 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e3f48a-ql8qx" Nov 03 21:00:44.491343 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:44.490920 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c034f31-xqldb" event={"ID":"6045b8c9-a516-4e66-8421-62fe0d7fbc79","Type":"ContainerStarted","Data":"614f822ce753f32931aff59de48a464fabcd99f3661ebe3d36a4860cd3f389c7"} Nov 03 21:00:44.491343 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:44.490979 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c034f31-xqldb" event={"ID":"6045b8c9-a516-4e66-8421-62fe0d7fbc79","Type":"ContainerStarted","Data":"e821330cab246528cea2c19883521e104f907ef261ee4b60da0aaf64dcd46d8c"} Nov 03 21:00:44.624522 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:44.622878 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c034f31-xqldb" podStartSLOduration=1.6228619690000001 podStartE2EDuration="1.622861969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:44.617586802 +0000 UTC m=+1279.842906592" watchObservedRunningTime="2026-11-03 21:00:44.622861969 +0000 UTC m=+1279.848181759" Nov 03 21:00:45.189356 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:45.189326 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd" Nov 03 21:00:45.190062 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:45.189753 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd" Nov 03 21:00:45.190994 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:45.190973 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd" Nov 03 21:00:45.218379 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:45.218344 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d" Nov 03 21:00:45.218941 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:45.218903 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d" Nov 03 21:00:45.220171 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:45.220150 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d" Nov 03 21:00:45.375092 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:45.375023 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd" Nov 03 21:00:45.375421 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:45.375405 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd" Nov 03 21:00:45.376068 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:45.376051 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd" Nov 03 21:00:45.502080 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:45.502039 4305 generic.go:356] "Generic (PLEG): container finished" podID="6045b8c9-a516-4e66-8421-62fe0d7fbc79" containerID="614f822ce753f32931aff59de48a464fabcd99f3661ebe3d36a4860cd3f389c7" exitCode=0 Nov 03 21:00:45.502605 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:45.502133 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c034f31-xqldb" event={"ID":"6045b8c9-a516-4e66-8421-62fe0d7fbc79","Type":"ContainerDied","Data":"614f822ce753f32931aff59de48a464fabcd99f3661ebe3d36a4860cd3f389c7"} Nov 03 21:00:45.552376 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:45.552341 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1c19d30-jt2b8"] Nov 03 21:00:45.553079 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:45.553051 4305 cpu_manager.go:414] "containerMap: removing container" podUID="12b59755-1aa9-4448-b884-cc45ad03be96" containerName="revoke-keys" Nov 03 21:00:45.553079 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:45.553078 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12b59755-1aa9-4448-b884-cc45ad03be96" containerName="revoke-keys" Nov 03 21:00:45.553271 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:45.553256 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="12b59755-1aa9-4448-b884-cc45ad03be96" containerName="revoke-keys" Nov 03 21:00:45.556944 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:45.556923 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1c19d30-jt2b8" Nov 03 21:00:45.614419 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:45.614389 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1c19d30-jt2b8"] Nov 03 21:00:45.679633 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:45.679591 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1c41cc7-b059-4ce5-9c82-f02e0d0f23c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1c19d30-jt2b8\" (UID: \"a1c41cc7-b059-4ce5-9c82-f02e0d0f23c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1c19d30-jt2b8" Nov 03 21:00:45.679840 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:45.679663 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1c41cc7-b059-4ce5-9c82-f02e0d0f23c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1c19d30-jt2b8\" (UID: \"a1c41cc7-b059-4ce5-9c82-f02e0d0f23c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1c19d30-jt2b8" Nov 03 21:00:45.679840 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:45.679723 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtvpd\" (UniqueName: \"kubernetes.io/projected/a1c41cc7-b059-4ce5-9c82-f02e0d0f23c0-kube-api-access-rtvpd\") pod \"maas-api-revoke-sub-b1c19d30-jt2b8\" (UID: \"a1c41cc7-b059-4ce5-9c82-f02e0d0f23c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1c19d30-jt2b8" Nov 03 21:00:45.781571 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:45.781535 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1c41cc7-b059-4ce5-9c82-f02e0d0f23c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1c19d30-jt2b8\" (UID: \"a1c41cc7-b059-4ce5-9c82-f02e0d0f23c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1c19d30-jt2b8" Nov 03 21:00:45.781738 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:45.781642 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1c41cc7-b059-4ce5-9c82-f02e0d0f23c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1c19d30-jt2b8\" (UID: \"a1c41cc7-b059-4ce5-9c82-f02e0d0f23c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1c19d30-jt2b8" Nov 03 21:00:45.781738 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:45.781689 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtvpd\" (UniqueName: \"kubernetes.io/projected/a1c41cc7-b059-4ce5-9c82-f02e0d0f23c0-kube-api-access-rtvpd\") pod \"maas-api-revoke-sub-b1c19d30-jt2b8\" (UID: \"a1c41cc7-b059-4ce5-9c82-f02e0d0f23c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1c19d30-jt2b8" Nov 03 21:00:45.782346 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:45.782324 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1c41cc7-b059-4ce5-9c82-f02e0d0f23c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1c19d30-jt2b8\" (UID: \"a1c41cc7-b059-4ce5-9c82-f02e0d0f23c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1c19d30-jt2b8" Nov 03 21:00:45.784087 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:45.784068 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1c41cc7-b059-4ce5-9c82-f02e0d0f23c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1c19d30-jt2b8\" (UID: \"a1c41cc7-b059-4ce5-9c82-f02e0d0f23c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1c19d30-jt2b8" Nov 03 21:00:45.870380 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:45.870342 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtvpd\" (UniqueName: \"kubernetes.io/projected/a1c41cc7-b059-4ce5-9c82-f02e0d0f23c0-kube-api-access-rtvpd\") pod \"maas-api-revoke-sub-b1c19d30-jt2b8\" (UID: \"a1c41cc7-b059-4ce5-9c82-f02e0d0f23c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1c19d30-jt2b8" Nov 03 21:00:45.870655 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:45.870639 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1c19d30-jt2b8" Nov 03 21:00:46.282483 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:46.282454 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1c19d30-jt2b8"] Nov 03 21:00:46.325351 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:00:46.325320 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:00:46.512676 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:46.512638 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1c19d30-jt2b8" event={"ID":"a1c41cc7-b059-4ce5-9c82-f02e0d0f23c0","Type":"ContainerStarted","Data":"9b1712bf12bf3faaf286937042c0a15770d75565ac342dc9412a0a66fd8b9a31"} Nov 03 21:00:46.656606 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:46.656576 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c034f31-xqldb" Nov 03 21:00:46.800413 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:46.800385 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6045b8c9-a516-4e66-8421-62fe0d7fbc79-kube-api-access-r7nwk\" (UniqueName: \"kubernetes.io/projected/6045b8c9-a516-4e66-8421-62fe0d7fbc79-kube-api-access-r7nwk\") pod \"6045b8c9-a516-4e66-8421-62fe0d7fbc79\" (UID: \"6045b8c9-a516-4e66-8421-62fe0d7fbc79\") " Nov 03 21:00:46.800582 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:46.800567 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6045b8c9-a516-4e66-8421-62fe0d7fbc79-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6045b8c9-a516-4e66-8421-62fe0d7fbc79-maas-api-openshift-ca\") pod \"6045b8c9-a516-4e66-8421-62fe0d7fbc79\" (UID: \"6045b8c9-a516-4e66-8421-62fe0d7fbc79\") " Nov 03 21:00:46.800654 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:46.800606 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6045b8c9-a516-4e66-8421-62fe0d7fbc79-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6045b8c9-a516-4e66-8421-62fe0d7fbc79-maas-api-xks-ca\") pod \"6045b8c9-a516-4e66-8421-62fe0d7fbc79\" (UID: \"6045b8c9-a516-4e66-8421-62fe0d7fbc79\") " Nov 03 21:00:46.801838 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:46.801811 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6045b8c9-a516-4e66-8421-62fe0d7fbc79-maas-api-openshift-ca" pod "6045b8c9-a516-4e66-8421-62fe0d7fbc79" (UID: "6045b8c9-a516-4e66-8421-62fe0d7fbc79"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:00:46.803223 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:46.803204 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6045b8c9-a516-4e66-8421-62fe0d7fbc79-maas-api-xks-ca" pod "6045b8c9-a516-4e66-8421-62fe0d7fbc79" (UID: "6045b8c9-a516-4e66-8421-62fe0d7fbc79"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:00:46.803297 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:46.803251 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6045b8c9-a516-4e66-8421-62fe0d7fbc79-kube-api-access-r7nwk" pod "6045b8c9-a516-4e66-8421-62fe0d7fbc79" (UID: "6045b8c9-a516-4e66-8421-62fe0d7fbc79"). InnerVolumeSpecName "kube-api-access-r7nwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:46.902538 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:46.902509 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7nwk\" (UniqueName: \"kubernetes.io/projected/6045b8c9-a516-4e66-8421-62fe0d7fbc79-kube-api-access-r7nwk\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:00:46.902538 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:46.902536 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6045b8c9-a516-4e66-8421-62fe0d7fbc79-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:00:46.902754 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:46.902549 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6045b8c9-a516-4e66-8421-62fe0d7fbc79-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:00:47.522762 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:47.522674 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c034f31-xqldb" Nov 03 21:00:47.522762 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:47.522705 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c034f31-xqldb" event={"ID":"6045b8c9-a516-4e66-8421-62fe0d7fbc79","Type":"ContainerDied","Data":"e821330cab246528cea2c19883521e104f907ef261ee4b60da0aaf64dcd46d8c"} Nov 03 21:00:47.523258 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:47.522774 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e821330cab246528cea2c19883521e104f907ef261ee4b60da0aaf64dcd46d8c" Nov 03 21:00:47.524608 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:47.524579 4305 generic.go:356] "Generic (PLEG): container finished" podID="a1c41cc7-b059-4ce5-9c82-f02e0d0f23c0" containerID="5a7e9a46ee4461e374db17845ca5c132fa94c0b74d53ea9f6345da8b67c08b44" exitCode=0 Nov 03 21:00:47.524760 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:47.524655 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1c19d30-jt2b8" event={"ID":"a1c41cc7-b059-4ce5-9c82-f02e0d0f23c0","Type":"ContainerDied","Data":"5a7e9a46ee4461e374db17845ca5c132fa94c0b74d53ea9f6345da8b67c08b44"} Nov 03 21:00:48.699839 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:48.699810 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1c19d30-jt2b8" Nov 03 21:00:48.833621 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:48.833539 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1c41cc7-b059-4ce5-9c82-f02e0d0f23c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1c41cc7-b059-4ce5-9c82-f02e0d0f23c0-maas-api-openshift-ca\") pod \"a1c41cc7-b059-4ce5-9c82-f02e0d0f23c0\" (UID: \"a1c41cc7-b059-4ce5-9c82-f02e0d0f23c0\") " Nov 03 21:00:48.833776 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:48.833617 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1c41cc7-b059-4ce5-9c82-f02e0d0f23c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1c41cc7-b059-4ce5-9c82-f02e0d0f23c0-maas-api-xks-ca\") pod \"a1c41cc7-b059-4ce5-9c82-f02e0d0f23c0\" (UID: \"a1c41cc7-b059-4ce5-9c82-f02e0d0f23c0\") " Nov 03 21:00:48.833776 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:48.833708 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1c41cc7-b059-4ce5-9c82-f02e0d0f23c0-kube-api-access-rtvpd\" (UniqueName: \"kubernetes.io/projected/a1c41cc7-b059-4ce5-9c82-f02e0d0f23c0-kube-api-access-rtvpd\") pod \"a1c41cc7-b059-4ce5-9c82-f02e0d0f23c0\" (UID: \"a1c41cc7-b059-4ce5-9c82-f02e0d0f23c0\") " Nov 03 21:00:48.833952 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:48.833924 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1c41cc7-b059-4ce5-9c82-f02e0d0f23c0-maas-api-openshift-ca" pod "a1c41cc7-b059-4ce5-9c82-f02e0d0f23c0" (UID: "a1c41cc7-b059-4ce5-9c82-f02e0d0f23c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:00:48.834670 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:48.834649 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1c41cc7-b059-4ce5-9c82-f02e0d0f23c0-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:00:48.836468 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:48.836423 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1c41cc7-b059-4ce5-9c82-f02e0d0f23c0-maas-api-xks-ca" pod "a1c41cc7-b059-4ce5-9c82-f02e0d0f23c0" (UID: "a1c41cc7-b059-4ce5-9c82-f02e0d0f23c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:00:48.836591 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:48.836471 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1c41cc7-b059-4ce5-9c82-f02e0d0f23c0-kube-api-access-rtvpd" pod "a1c41cc7-b059-4ce5-9c82-f02e0d0f23c0" (UID: "a1c41cc7-b059-4ce5-9c82-f02e0d0f23c0"). InnerVolumeSpecName "kube-api-access-rtvpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:48.935936 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:48.935893 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1c41cc7-b059-4ce5-9c82-f02e0d0f23c0-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:00:48.935936 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:48.935929 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtvpd\" (UniqueName: \"kubernetes.io/projected/a1c41cc7-b059-4ce5-9c82-f02e0d0f23c0-kube-api-access-rtvpd\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:00:49.542230 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:49.542195 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1c19d30-jt2b8" Nov 03 21:00:49.542459 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:49.542230 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1c19d30-jt2b8" event={"ID":"a1c41cc7-b059-4ce5-9c82-f02e0d0f23c0","Type":"ContainerDied","Data":"9b1712bf12bf3faaf286937042c0a15770d75565ac342dc9412a0a66fd8b9a31"} Nov 03 21:00:49.542459 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:49.542278 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b1712bf12bf3faaf286937042c0a15770d75565ac342dc9412a0a66fd8b9a31" Nov 03 21:00:55.594580 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:55.594552 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:00:56.390729 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:56.390695 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b37ba63-sfzx2"] Nov 03 21:00:56.392499 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:56.391955 4305 cpu_manager.go:414] "containerMap: removing container" podUID="6045b8c9-a516-4e66-8421-62fe0d7fbc79" containerName="revoke-keys" Nov 03 21:00:56.403457 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:56.399519 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6045b8c9-a516-4e66-8421-62fe0d7fbc79" containerName="revoke-keys" Nov 03 21:00:56.403457 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:56.399596 4305 cpu_manager.go:414] "containerMap: removing container" podUID="a1c41cc7-b059-4ce5-9c82-f02e0d0f23c0" containerName="revoke-keys" Nov 03 21:00:56.403457 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:56.399607 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1c41cc7-b059-4ce5-9c82-f02e0d0f23c0" containerName="revoke-keys" Nov 03 21:00:56.403457 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:56.399852 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="6045b8c9-a516-4e66-8421-62fe0d7fbc79" containerName="revoke-keys" Nov 03 21:00:56.403457 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:56.399873 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1c41cc7-b059-4ce5-9c82-f02e0d0f23c0" containerName="revoke-keys" Nov 03 21:00:56.412694 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:56.412668 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b37ba63-sfzx2"] Nov 03 21:00:56.412920 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:56.412794 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b37ba63-sfzx2" Nov 03 21:00:56.481802 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:56.480476 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:00:56.575454 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:56.568992 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77vtd\"" Nov 03 21:00:56.575454 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:56.569114 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:00:56.575454 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:56.569233 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:00:56.582456 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:56.581371 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5cc19fe-8df7-4ccb-9cc6-d9c0fd429a27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b37ba63-sfzx2\" (UID: \"b5cc19fe-8df7-4ccb-9cc6-d9c0fd429a27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b37ba63-sfzx2" Nov 03 21:00:56.582456 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:56.581508 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5cc19fe-8df7-4ccb-9cc6-d9c0fd429a27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b37ba63-sfzx2\" (UID: \"b5cc19fe-8df7-4ccb-9cc6-d9c0fd429a27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b37ba63-sfzx2" Nov 03 21:00:56.582456 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:56.581674 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrh6d\" (UniqueName: \"kubernetes.io/projected/b5cc19fe-8df7-4ccb-9cc6-d9c0fd429a27-kube-api-access-lrh6d\") pod \"maas-api-revoke-sub-1b37ba63-sfzx2\" (UID: \"b5cc19fe-8df7-4ccb-9cc6-d9c0fd429a27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b37ba63-sfzx2" Nov 03 21:00:56.684299 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:56.684246 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5cc19fe-8df7-4ccb-9cc6-d9c0fd429a27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b37ba63-sfzx2\" (UID: \"b5cc19fe-8df7-4ccb-9cc6-d9c0fd429a27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b37ba63-sfzx2" Nov 03 21:00:56.685995 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:56.685971 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrh6d\" (UniqueName: \"kubernetes.io/projected/b5cc19fe-8df7-4ccb-9cc6-d9c0fd429a27-kube-api-access-lrh6d\") pod \"maas-api-revoke-sub-1b37ba63-sfzx2\" (UID: \"b5cc19fe-8df7-4ccb-9cc6-d9c0fd429a27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b37ba63-sfzx2" Nov 03 21:00:56.686629 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:56.686586 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5cc19fe-8df7-4ccb-9cc6-d9c0fd429a27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b37ba63-sfzx2\" (UID: \"b5cc19fe-8df7-4ccb-9cc6-d9c0fd429a27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b37ba63-sfzx2" Nov 03 21:00:56.687220 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:56.685782 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5cc19fe-8df7-4ccb-9cc6-d9c0fd429a27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b37ba63-sfzx2\" (UID: \"b5cc19fe-8df7-4ccb-9cc6-d9c0fd429a27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b37ba63-sfzx2" Nov 03 21:00:56.712564 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:56.712488 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5cc19fe-8df7-4ccb-9cc6-d9c0fd429a27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b37ba63-sfzx2\" (UID: \"b5cc19fe-8df7-4ccb-9cc6-d9c0fd429a27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b37ba63-sfzx2" Nov 03 21:00:56.752240 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:56.752171 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrh6d\" (UniqueName: \"kubernetes.io/projected/b5cc19fe-8df7-4ccb-9cc6-d9c0fd429a27-kube-api-access-lrh6d\") pod \"maas-api-revoke-sub-1b37ba63-sfzx2\" (UID: \"b5cc19fe-8df7-4ccb-9cc6-d9c0fd429a27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b37ba63-sfzx2" Nov 03 21:00:56.941166 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:56.936938 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b37ba63-sfzx2" Nov 03 21:00:57.982460 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:57.977601 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" podUID="793b8ca3-3a2e-49a1-9be4-4971ba59df09" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 03 21:00:58.081795 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:58.012749 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" podUID="d5d3dcaa-a51a-40c6-8049-871b300552e1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:00:58.081795 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:58.079566 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" podUID="80de7944-70f6-49e1-a0e8-4b82d5caad0d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:00:58.186615 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:58.168596 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" podUID="6e7f30a9-0862-48a5-9ab7-e692e75bb294" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:00:58.254541 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:58.253877 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" podUID="7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:00:58.343542 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:58.280080 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" podUID="8ee1f028-8503-4034-9954-eb06fcf402e7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": context deadline exceeded" Nov 03 21:00:58.433553 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:58.432482 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" podUID="63a704b4-51ce-46c3-8007-34b308aff332" containerName="main" probeResult="failure" output="Get \"https://10.134.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:00:58.497491 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:58.484571 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" podUID="7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 03 21:00:58.650530 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:58.558403 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" podUID="006c5eae-06af-447b-a1d8-cd719cce2a92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Nov 03 21:00:58.650530 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:58.632983 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" podUID="8793f095-97b1-4e0a-8382-b541f90a0de2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:00:58.841461 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:58.750010 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" podUID="be06bc96-808a-46e6-9872-01baf1a11a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:00:58.980343 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:58.980299 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" podUID="793b8ca3-3a2e-49a1-9be4-4971ba59df09" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:00:59.019670 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:59.019607 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" podUID="d5d3dcaa-a51a-40c6-8049-871b300552e1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:00:59.124749 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:59.124701 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" podUID="80de7944-70f6-49e1-a0e8-4b82d5caad0d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:00:59.261950 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:59.198540 4305 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" podUID="793b8ca3-3a2e-49a1-9be4-4971ba59df09" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": context deadline exceeded" Nov 03 21:00:59.262363 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:59.214785 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" podUID="6e7f30a9-0862-48a5-9ab7-e692e75bb294" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 03 21:00:59.262704 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:59.254570 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" podUID="7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:00:59.306496 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:59.280701 4305 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" podUID="6e7f30a9-0862-48a5-9ab7-e692e75bb294" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:00:59.306496 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:59.281594 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" podUID="8ee1f028-8503-4034-9954-eb06fcf402e7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:00:59.433593 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:59.433550 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" podUID="63a704b4-51ce-46c3-8007-34b308aff332" containerName="main" probeResult="failure" output="Get \"https://10.134.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:00:59.508635 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:59.485568 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" podUID="7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:00:59.512989 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:59.512913 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" Nov 03 21:00:59.589557 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:59.589525 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" Nov 03 21:00:59.589839 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:59.589822 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" Nov 03 21:00:59.589990 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:59.589977 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" Nov 03 21:00:59.597549 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:59.597517 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b37ba63-sfzx2"] Nov 03 21:00:59.659512 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:59.634309 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" Nov 03 21:00:59.667617 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:59.667583 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" Nov 03 21:00:59.824691 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:59.559576 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" podUID="006c5eae-06af-447b-a1d8-cd719cce2a92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:00:59.825293 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:59.702204 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" podUID="8793f095-97b1-4e0a-8382-b541f90a0de2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:00:59.825627 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:59.702239 4305 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" podUID="be06bc96-808a-46e6-9872-01baf1a11a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.107:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:00:59.825789 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:59.801012 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:00:59.831467 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:59.825888 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s" podUID="c7f2f4c8-63bb-4a07-8860-91a420ebc1ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:00:59.892158 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:59.892104 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" podUID="be06bc96-808a-46e6-9872-01baf1a11a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:00:59.895509 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:59.832967 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b37ba63-sfzx2" event={"ID":"b5cc19fe-8df7-4ccb-9cc6-d9c0fd429a27","Type":"ContainerStarted","Data":"798c440b6a2ff0077bb8d6f8c7636ecc1e7ececa730ac7ed7240c10ee8bd443e"} Nov 03 21:00:59.895780 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:59.895763 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" Nov 03 21:00:59.895920 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:59.895909 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" Nov 03 21:00:59.896096 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:59.896083 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" Nov 03 21:00:59.896249 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:59.896237 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" Nov 03 21:00:59.905791 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:59.905761 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" Nov 03 21:00:59.997775 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:59.995649 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" Nov 03 21:00:59.997775 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:00:59.996325 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" podUID="793b8ca3-3a2e-49a1-9be4-4971ba59df09" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:00.016559 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:00.016367 4305 patch_prober.go:29] interesting pod/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:01:00.016559 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:00.016473 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc" podUID="6a908050-5ca3-4977-b279-67965ac30de1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:00.033550 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:00.033508 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" podUID="d5d3dcaa-a51a-40c6-8049-871b300552e1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:00.042875 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:00.042848 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" Nov 03 21:01:00.057452 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:00.056303 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" Nov 03 21:01:00.069097 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:00.065571 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-659eab4c-drdgf"] Nov 03 21:01:00.073978 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:00.073953 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" Nov 03 21:01:00.113864 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:00.092595 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" Nov 03 21:01:00.113864 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:00.104508 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-659eab4c-drdgf"] Nov 03 21:01:00.162482 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:00.155726 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" Nov 03 21:01:00.162482 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:00.156264 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" Nov 03 21:01:00.162482 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:00.159751 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" Nov 03 21:01:00.170305 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:00.164128 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" Nov 03 21:01:00.170305 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:00.169897 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" Nov 03 21:01:00.170305 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:00.169948 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" Nov 03 21:01:00.792275 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:00.792236 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b37ba63-sfzx2" event={"ID":"b5cc19fe-8df7-4ccb-9cc6-d9c0fd429a27","Type":"ContainerStarted","Data":"d2b0f0ed4c2715033c8d86fa3586aedabf80d00b1ed36b12ecedd9556356aeeb"} Nov 03 21:01:00.897754 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:00.897721 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48077494-2ea6-4d8f-aab2-32afc6872ae1" path="/var/lib/kubelet/pods/48077494-2ea6-4d8f-aab2-32afc6872ae1/volumes" Nov 03 21:01:01.805389 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:01.805353 4305 generic.go:356] "Generic (PLEG): container finished" podID="b5cc19fe-8df7-4ccb-9cc6-d9c0fd429a27" containerID="d2b0f0ed4c2715033c8d86fa3586aedabf80d00b1ed36b12ecedd9556356aeeb" exitCode=0 Nov 03 21:01:01.805846 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:01.805456 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b37ba63-sfzx2" event={"ID":"b5cc19fe-8df7-4ccb-9cc6-d9c0fd429a27","Type":"ContainerDied","Data":"d2b0f0ed4c2715033c8d86fa3586aedabf80d00b1ed36b12ecedd9556356aeeb"} Nov 03 21:01:01.866880 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:01.866824 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b37ba63-sfzx2" podStartSLOduration=5.86680888 podStartE2EDuration="5.86680888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:01:01.863692338 +0000 UTC m=+1297.089012149" watchObservedRunningTime="2026-11-03 21:01:01.86680888 +0000 UTC m=+1297.092128740" Nov 03 21:01:02.265406 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:02.265377 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2778bcd-v2xb6"] Nov 03 21:01:02.269668 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:02.269647 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2778bcd-v2xb6" Nov 03 21:01:02.318013 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:02.317975 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6scp6\" (UniqueName: \"kubernetes.io/projected/5cbd2172-86f0-440f-9b21-ba958b56af86-kube-api-access-6scp6\") pod \"maas-api-revoke-sub-f2778bcd-v2xb6\" (UID: \"5cbd2172-86f0-440f-9b21-ba958b56af86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2778bcd-v2xb6" Nov 03 21:01:02.318167 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:02.318124 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cbd2172-86f0-440f-9b21-ba958b56af86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2778bcd-v2xb6\" (UID: \"5cbd2172-86f0-440f-9b21-ba958b56af86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2778bcd-v2xb6" Nov 03 21:01:02.318228 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:02.318213 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cbd2172-86f0-440f-9b21-ba958b56af86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2778bcd-v2xb6\" (UID: \"5cbd2172-86f0-440f-9b21-ba958b56af86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2778bcd-v2xb6" Nov 03 21:01:02.327832 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:02.327799 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2778bcd-v2xb6"] Nov 03 21:01:02.419913 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:02.419868 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6scp6\" (UniqueName: \"kubernetes.io/projected/5cbd2172-86f0-440f-9b21-ba958b56af86-kube-api-access-6scp6\") pod \"maas-api-revoke-sub-f2778bcd-v2xb6\" (UID: \"5cbd2172-86f0-440f-9b21-ba958b56af86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2778bcd-v2xb6" Nov 03 21:01:02.420122 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:02.420017 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cbd2172-86f0-440f-9b21-ba958b56af86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2778bcd-v2xb6\" (UID: \"5cbd2172-86f0-440f-9b21-ba958b56af86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2778bcd-v2xb6" Nov 03 21:01:02.420122 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:02.420110 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cbd2172-86f0-440f-9b21-ba958b56af86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2778bcd-v2xb6\" (UID: \"5cbd2172-86f0-440f-9b21-ba958b56af86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2778bcd-v2xb6" Nov 03 21:01:02.420819 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:02.420799 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cbd2172-86f0-440f-9b21-ba958b56af86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2778bcd-v2xb6\" (UID: \"5cbd2172-86f0-440f-9b21-ba958b56af86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2778bcd-v2xb6" Nov 03 21:01:02.422666 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:02.422644 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cbd2172-86f0-440f-9b21-ba958b56af86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2778bcd-v2xb6\" (UID: \"5cbd2172-86f0-440f-9b21-ba958b56af86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2778bcd-v2xb6" Nov 03 21:01:02.481015 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:02.480983 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6scp6\" (UniqueName: \"kubernetes.io/projected/5cbd2172-86f0-440f-9b21-ba958b56af86-kube-api-access-6scp6\") pod \"maas-api-revoke-sub-f2778bcd-v2xb6\" (UID: \"5cbd2172-86f0-440f-9b21-ba958b56af86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2778bcd-v2xb6" Nov 03 21:01:02.584160 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:02.584086 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2778bcd-v2xb6" Nov 03 21:01:02.965910 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:02.965885 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b37ba63-sfzx2" Nov 03 21:01:02.968776 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:01:02.968746 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cbd2172_86f0_440f_9b21_ba958b56af86.slice/crio-fbb1206ca520ee099a93e77e3e99d14b96d33ba7f68bd6a531fc4cbdeb03acfd WatchSource:0}: Error finding container fbb1206ca520ee099a93e77e3e99d14b96d33ba7f68bd6a531fc4cbdeb03acfd: Status 404 returned error can't find the container with id fbb1206ca520ee099a93e77e3e99d14b96d33ba7f68bd6a531fc4cbdeb03acfd Nov 03 21:01:02.969797 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:02.969772 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2778bcd-v2xb6"] Nov 03 21:01:03.029497 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:03.029428 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5cc19fe-8df7-4ccb-9cc6-d9c0fd429a27-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5cc19fe-8df7-4ccb-9cc6-d9c0fd429a27-maas-api-xks-ca\") pod \"b5cc19fe-8df7-4ccb-9cc6-d9c0fd429a27\" (UID: \"b5cc19fe-8df7-4ccb-9cc6-d9c0fd429a27\") " Nov 03 21:01:03.029688 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:03.029555 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5cc19fe-8df7-4ccb-9cc6-d9c0fd429a27-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5cc19fe-8df7-4ccb-9cc6-d9c0fd429a27-maas-api-openshift-ca\") pod \"b5cc19fe-8df7-4ccb-9cc6-d9c0fd429a27\" (UID: \"b5cc19fe-8df7-4ccb-9cc6-d9c0fd429a27\") " Nov 03 21:01:03.029688 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:03.029677 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5cc19fe-8df7-4ccb-9cc6-d9c0fd429a27-kube-api-access-lrh6d\" (UniqueName: \"kubernetes.io/projected/b5cc19fe-8df7-4ccb-9cc6-d9c0fd429a27-kube-api-access-lrh6d\") pod \"b5cc19fe-8df7-4ccb-9cc6-d9c0fd429a27\" (UID: \"b5cc19fe-8df7-4ccb-9cc6-d9c0fd429a27\") " Nov 03 21:01:03.029949 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:03.029925 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5cc19fe-8df7-4ccb-9cc6-d9c0fd429a27-maas-api-openshift-ca" pod "b5cc19fe-8df7-4ccb-9cc6-d9c0fd429a27" (UID: "b5cc19fe-8df7-4ccb-9cc6-d9c0fd429a27"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:01:03.030239 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:03.030222 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5cc19fe-8df7-4ccb-9cc6-d9c0fd429a27-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:01:03.031604 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:03.031585 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5cc19fe-8df7-4ccb-9cc6-d9c0fd429a27-maas-api-xks-ca" pod "b5cc19fe-8df7-4ccb-9cc6-d9c0fd429a27" (UID: "b5cc19fe-8df7-4ccb-9cc6-d9c0fd429a27"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:01:03.031684 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:03.031664 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5cc19fe-8df7-4ccb-9cc6-d9c0fd429a27-kube-api-access-lrh6d" pod "b5cc19fe-8df7-4ccb-9cc6-d9c0fd429a27" (UID: "b5cc19fe-8df7-4ccb-9cc6-d9c0fd429a27"). InnerVolumeSpecName "kube-api-access-lrh6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:01:03.131700 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:03.131616 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrh6d\" (UniqueName: \"kubernetes.io/projected/b5cc19fe-8df7-4ccb-9cc6-d9c0fd429a27-kube-api-access-lrh6d\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:01:03.131700 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:03.131649 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5cc19fe-8df7-4ccb-9cc6-d9c0fd429a27-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:01:03.824537 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:03.824499 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2778bcd-v2xb6" event={"ID":"5cbd2172-86f0-440f-9b21-ba958b56af86","Type":"ContainerStarted","Data":"ce40284ca2fc4ee9e0a14a312855808de1f3988f4e0e213c0ab74e921fd420f8"} Nov 03 21:01:03.824879 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:03.824572 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2778bcd-v2xb6" event={"ID":"5cbd2172-86f0-440f-9b21-ba958b56af86","Type":"ContainerStarted","Data":"fbb1206ca520ee099a93e77e3e99d14b96d33ba7f68bd6a531fc4cbdeb03acfd"} Nov 03 21:01:03.826499 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:03.826449 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b37ba63-sfzx2" event={"ID":"b5cc19fe-8df7-4ccb-9cc6-d9c0fd429a27","Type":"ContainerDied","Data":"798c440b6a2ff0077bb8d6f8c7636ecc1e7ececa730ac7ed7240c10ee8bd443e"} Nov 03 21:01:03.826499 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:03.826490 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b37ba63-sfzx2" Nov 03 21:01:03.826742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:03.826505 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="798c440b6a2ff0077bb8d6f8c7636ecc1e7ececa730ac7ed7240c10ee8bd443e" Nov 03 21:01:03.940389 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:03.940314 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2778bcd-v2xb6" podStartSLOduration=1.940294004 podStartE2EDuration="1.940294004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:01:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:01:03.939219299 +0000 UTC m=+1299.164539110" watchObservedRunningTime="2026-11-03 21:01:03.940294004 +0000 UTC m=+1299.165613796" Nov 03 21:01:04.835834 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:04.835796 4305 generic.go:356] "Generic (PLEG): container finished" podID="5cbd2172-86f0-440f-9b21-ba958b56af86" containerID="ce40284ca2fc4ee9e0a14a312855808de1f3988f4e0e213c0ab74e921fd420f8" exitCode=0 Nov 03 21:01:04.836251 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:04.835876 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2778bcd-v2xb6" event={"ID":"5cbd2172-86f0-440f-9b21-ba958b56af86","Type":"ContainerDied","Data":"ce40284ca2fc4ee9e0a14a312855808de1f3988f4e0e213c0ab74e921fd420f8"} Nov 03 21:01:06.012504 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:06.012480 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2778bcd-v2xb6" Nov 03 21:01:06.073899 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:06.073868 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cbd2172-86f0-440f-9b21-ba958b56af86-kube-api-access-6scp6\" (UniqueName: \"kubernetes.io/projected/5cbd2172-86f0-440f-9b21-ba958b56af86-kube-api-access-6scp6\") pod \"5cbd2172-86f0-440f-9b21-ba958b56af86\" (UID: \"5cbd2172-86f0-440f-9b21-ba958b56af86\") " Nov 03 21:01:06.074092 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:06.074042 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cbd2172-86f0-440f-9b21-ba958b56af86-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cbd2172-86f0-440f-9b21-ba958b56af86-maas-api-xks-ca\") pod \"5cbd2172-86f0-440f-9b21-ba958b56af86\" (UID: \"5cbd2172-86f0-440f-9b21-ba958b56af86\") " Nov 03 21:01:06.074092 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:06.074065 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cbd2172-86f0-440f-9b21-ba958b56af86-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cbd2172-86f0-440f-9b21-ba958b56af86-maas-api-openshift-ca\") pod \"5cbd2172-86f0-440f-9b21-ba958b56af86\" (UID: \"5cbd2172-86f0-440f-9b21-ba958b56af86\") " Nov 03 21:01:06.074539 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:06.074505 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cbd2172-86f0-440f-9b21-ba958b56af86-maas-api-openshift-ca" pod "5cbd2172-86f0-440f-9b21-ba958b56af86" (UID: "5cbd2172-86f0-440f-9b21-ba958b56af86"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:01:06.074752 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:06.074733 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cbd2172-86f0-440f-9b21-ba958b56af86-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:01:06.076263 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:06.076238 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cbd2172-86f0-440f-9b21-ba958b56af86-kube-api-access-6scp6" pod "5cbd2172-86f0-440f-9b21-ba958b56af86" (UID: "5cbd2172-86f0-440f-9b21-ba958b56af86"). InnerVolumeSpecName "kube-api-access-6scp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:01:06.076341 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:06.076301 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cbd2172-86f0-440f-9b21-ba958b56af86-maas-api-xks-ca" pod "5cbd2172-86f0-440f-9b21-ba958b56af86" (UID: "5cbd2172-86f0-440f-9b21-ba958b56af86"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:01:06.176329 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:06.176228 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cbd2172-86f0-440f-9b21-ba958b56af86-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:01:06.176329 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:06.176267 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6scp6\" (UniqueName: \"kubernetes.io/projected/5cbd2172-86f0-440f-9b21-ba958b56af86-kube-api-access-6scp6\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:01:06.852497 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:06.852463 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2778bcd-v2xb6" event={"ID":"5cbd2172-86f0-440f-9b21-ba958b56af86","Type":"ContainerDied","Data":"fbb1206ca520ee099a93e77e3e99d14b96d33ba7f68bd6a531fc4cbdeb03acfd"} Nov 03 21:01:06.852656 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:06.852515 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fbb1206ca520ee099a93e77e3e99d14b96d33ba7f68bd6a531fc4cbdeb03acfd" Nov 03 21:01:06.852656 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:06.852471 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2778bcd-v2xb6" Nov 03 21:01:07.051233 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:07.051203 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fca8d49-xqrvf"] Nov 03 21:01:07.055664 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:07.055641 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fca8d49-xqrvf"] Nov 03 21:01:08.896818 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:08.896784 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a15184d3-975f-4101-9823-3686d63d3f8d" path="/var/lib/kubelet/pods/a15184d3-975f-4101-9823-3686d63d3f8d/volumes" Nov 03 21:01:11.051786 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:11.051753 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-812c2c2c-h24x7"] Nov 03 21:01:11.101240 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:11.101202 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-812c2c2c-h24x7"] Nov 03 21:01:12.847548 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:12.847509 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0c0b95e-b5sjc"] Nov 03 21:01:12.848216 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:12.848201 4305 cpu_manager.go:414] "containerMap: removing container" podUID="b5cc19fe-8df7-4ccb-9cc6-d9c0fd429a27" containerName="revoke-keys" Nov 03 21:01:12.848262 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:12.848217 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5cc19fe-8df7-4ccb-9cc6-d9c0fd429a27" containerName="revoke-keys" Nov 03 21:01:12.848262 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:12.848254 4305 cpu_manager.go:414] "containerMap: removing container" podUID="5cbd2172-86f0-440f-9b21-ba958b56af86" containerName="revoke-keys" Nov 03 21:01:12.848262 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:12.848260 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cbd2172-86f0-440f-9b21-ba958b56af86" containerName="revoke-keys" Nov 03 21:01:12.848384 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:12.848374 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5cc19fe-8df7-4ccb-9cc6-d9c0fd429a27" containerName="revoke-keys" Nov 03 21:01:12.848424 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:12.848389 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cbd2172-86f0-440f-9b21-ba958b56af86" containerName="revoke-keys" Nov 03 21:01:12.853598 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:12.853576 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0c0b95e-b5sjc" Nov 03 21:01:12.897302 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:12.897268 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b3bcecb-831c-49d0-a4ee-72c965fe9872" path="/var/lib/kubelet/pods/1b3bcecb-831c-49d0-a4ee-72c965fe9872/volumes" Nov 03 21:01:12.909591 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:12.909553 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0c0b95e-b5sjc"] Nov 03 21:01:12.952624 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:12.952585 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:01:12.952624 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:12.952599 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:01:12.952841 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:12.952729 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:01:12.955314 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:12.955294 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77vtd\"" Nov 03 21:01:12.978525 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:12.978489 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v7ch\" (UniqueName: \"kubernetes.io/projected/ab9a9629-374b-45ef-a685-56da1ae8bfda-kube-api-access-9v7ch\") pod \"maas-api-revoke-sub-e0c0b95e-b5sjc\" (UID: \"ab9a9629-374b-45ef-a685-56da1ae8bfda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0c0b95e-b5sjc" Nov 03 21:01:12.978702 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:12.978557 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab9a9629-374b-45ef-a685-56da1ae8bfda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0c0b95e-b5sjc\" (UID: \"ab9a9629-374b-45ef-a685-56da1ae8bfda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0c0b95e-b5sjc" Nov 03 21:01:12.978824 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:12.978801 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab9a9629-374b-45ef-a685-56da1ae8bfda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0c0b95e-b5sjc\" (UID: \"ab9a9629-374b-45ef-a685-56da1ae8bfda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0c0b95e-b5sjc" Nov 03 21:01:13.080260 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:13.080225 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab9a9629-374b-45ef-a685-56da1ae8bfda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0c0b95e-b5sjc\" (UID: \"ab9a9629-374b-45ef-a685-56da1ae8bfda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0c0b95e-b5sjc" Nov 03 21:01:13.080500 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:13.080381 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v7ch\" (UniqueName: \"kubernetes.io/projected/ab9a9629-374b-45ef-a685-56da1ae8bfda-kube-api-access-9v7ch\") pod \"maas-api-revoke-sub-e0c0b95e-b5sjc\" (UID: \"ab9a9629-374b-45ef-a685-56da1ae8bfda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0c0b95e-b5sjc" Nov 03 21:01:13.080500 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:13.080467 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab9a9629-374b-45ef-a685-56da1ae8bfda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0c0b95e-b5sjc\" (UID: \"ab9a9629-374b-45ef-a685-56da1ae8bfda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0c0b95e-b5sjc" Nov 03 21:01:13.080944 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:13.080919 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab9a9629-374b-45ef-a685-56da1ae8bfda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0c0b95e-b5sjc\" (UID: \"ab9a9629-374b-45ef-a685-56da1ae8bfda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0c0b95e-b5sjc" Nov 03 21:01:13.083217 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:13.083197 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab9a9629-374b-45ef-a685-56da1ae8bfda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0c0b95e-b5sjc\" (UID: \"ab9a9629-374b-45ef-a685-56da1ae8bfda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0c0b95e-b5sjc" Nov 03 21:01:13.143127 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:13.143094 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v7ch\" (UniqueName: \"kubernetes.io/projected/ab9a9629-374b-45ef-a685-56da1ae8bfda-kube-api-access-9v7ch\") pod \"maas-api-revoke-sub-e0c0b95e-b5sjc\" (UID: \"ab9a9629-374b-45ef-a685-56da1ae8bfda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0c0b95e-b5sjc" Nov 03 21:01:13.167937 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:13.167902 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0c0b95e-b5sjc" Nov 03 21:01:13.561902 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:01:13.561871 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab9a9629_374b_45ef_a685_56da1ae8bfda.slice/crio-bd14c59fe944700120ce1aefa3c3fc29e963c7baf131aabf8305c8d52fb8dfa7 WatchSource:0}: Error finding container bd14c59fe944700120ce1aefa3c3fc29e963c7baf131aabf8305c8d52fb8dfa7: Status 404 returned error can't find the container with id bd14c59fe944700120ce1aefa3c3fc29e963c7baf131aabf8305c8d52fb8dfa7 Nov 03 21:01:13.562380 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:13.562352 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0c0b95e-b5sjc"] Nov 03 21:01:13.918019 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:13.917985 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0c0b95e-b5sjc" event={"ID":"ab9a9629-374b-45ef-a685-56da1ae8bfda","Type":"ContainerStarted","Data":"bd14c59fe944700120ce1aefa3c3fc29e963c7baf131aabf8305c8d52fb8dfa7"} Nov 03 21:01:14.928085 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:14.928049 4305 generic.go:356] "Generic (PLEG): container finished" podID="ab9a9629-374b-45ef-a685-56da1ae8bfda" containerID="f6f1b0e91566d64933a43114d277e1241c02a15e94cb0acac95e4c1bc6839a24" exitCode=0 Nov 03 21:01:14.928595 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:14.928133 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0c0b95e-b5sjc" event={"ID":"ab9a9629-374b-45ef-a685-56da1ae8bfda","Type":"ContainerDied","Data":"f6f1b0e91566d64933a43114d277e1241c02a15e94cb0acac95e4c1bc6839a24"} Nov 03 21:01:15.097930 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:15.097897 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0550170-xdbmz"] Nov 03 21:01:15.102483 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:15.102459 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0550170-xdbmz"] Nov 03 21:01:16.150232 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:16.150207 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0c0b95e-b5sjc" Nov 03 21:01:16.330816 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:16.330730 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab9a9629-374b-45ef-a685-56da1ae8bfda-kube-api-access-9v7ch\" (UniqueName: \"kubernetes.io/projected/ab9a9629-374b-45ef-a685-56da1ae8bfda-kube-api-access-9v7ch\") pod \"ab9a9629-374b-45ef-a685-56da1ae8bfda\" (UID: \"ab9a9629-374b-45ef-a685-56da1ae8bfda\") " Nov 03 21:01:16.330816 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:16.330774 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab9a9629-374b-45ef-a685-56da1ae8bfda-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab9a9629-374b-45ef-a685-56da1ae8bfda-maas-api-openshift-ca\") pod \"ab9a9629-374b-45ef-a685-56da1ae8bfda\" (UID: \"ab9a9629-374b-45ef-a685-56da1ae8bfda\") " Nov 03 21:01:16.331037 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:16.330970 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab9a9629-374b-45ef-a685-56da1ae8bfda-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab9a9629-374b-45ef-a685-56da1ae8bfda-maas-api-xks-ca\") pod \"ab9a9629-374b-45ef-a685-56da1ae8bfda\" (UID: \"ab9a9629-374b-45ef-a685-56da1ae8bfda\") " Nov 03 21:01:16.331128 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:01:16.331086 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:01:16.331215 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:16.331193 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab9a9629-374b-45ef-a685-56da1ae8bfda-maas-api-openshift-ca" pod "ab9a9629-374b-45ef-a685-56da1ae8bfda" (UID: "ab9a9629-374b-45ef-a685-56da1ae8bfda"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:01:16.331722 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:16.331617 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab9a9629-374b-45ef-a685-56da1ae8bfda-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:01:16.333161 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:16.333135 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab9a9629-374b-45ef-a685-56da1ae8bfda-maas-api-xks-ca" pod "ab9a9629-374b-45ef-a685-56da1ae8bfda" (UID: "ab9a9629-374b-45ef-a685-56da1ae8bfda"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:01:16.333252 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:16.333187 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab9a9629-374b-45ef-a685-56da1ae8bfda-kube-api-access-9v7ch" pod "ab9a9629-374b-45ef-a685-56da1ae8bfda" (UID: "ab9a9629-374b-45ef-a685-56da1ae8bfda"). InnerVolumeSpecName "kube-api-access-9v7ch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:01:16.433005 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:16.432966 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9v7ch\" (UniqueName: \"kubernetes.io/projected/ab9a9629-374b-45ef-a685-56da1ae8bfda-kube-api-access-9v7ch\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:01:16.433005 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:16.433000 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab9a9629-374b-45ef-a685-56da1ae8bfda-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:01:16.897230 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:16.897199 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bdd446d-01b5-40a3-869d-0e7199d7a64f" path="/var/lib/kubelet/pods/4bdd446d-01b5-40a3-869d-0e7199d7a64f/volumes" Nov 03 21:01:16.954052 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:16.954024 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0c0b95e-b5sjc" Nov 03 21:01:16.954238 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:16.954066 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0c0b95e-b5sjc" event={"ID":"ab9a9629-374b-45ef-a685-56da1ae8bfda","Type":"ContainerDied","Data":"bd14c59fe944700120ce1aefa3c3fc29e963c7baf131aabf8305c8d52fb8dfa7"} Nov 03 21:01:16.954238 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:16.954118 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd14c59fe944700120ce1aefa3c3fc29e963c7baf131aabf8305c8d52fb8dfa7" Nov 03 21:01:18.798177 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:18.798139 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31b20f9f-bbs4g"] Nov 03 21:01:18.799141 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:18.799122 4305 cpu_manager.go:414] "containerMap: removing container" podUID="ab9a9629-374b-45ef-a685-56da1ae8bfda" containerName="revoke-keys" Nov 03 21:01:18.799186 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:18.799144 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab9a9629-374b-45ef-a685-56da1ae8bfda" containerName="revoke-keys" Nov 03 21:01:18.799284 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:18.799274 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab9a9629-374b-45ef-a685-56da1ae8bfda" containerName="revoke-keys" Nov 03 21:01:18.803191 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:18.803171 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31b20f9f-bbs4g" Nov 03 21:01:18.855604 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:18.855570 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:01:18.855914 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:18.855875 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:01:18.862159 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:18.862129 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31b20f9f-bbs4g"] Nov 03 21:01:18.865657 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:18.865629 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2806e408-4be8-4af6-90b3-4c122a06b402-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31b20f9f-bbs4g\" (UID: \"2806e408-4be8-4af6-90b3-4c122a06b402\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31b20f9f-bbs4g" Nov 03 21:01:18.865819 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:18.865757 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2806e408-4be8-4af6-90b3-4c122a06b402-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31b20f9f-bbs4g\" (UID: \"2806e408-4be8-4af6-90b3-4c122a06b402\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31b20f9f-bbs4g" Nov 03 21:01:18.865900 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:18.865830 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvnsm\" (UniqueName: \"kubernetes.io/projected/2806e408-4be8-4af6-90b3-4c122a06b402-kube-api-access-lvnsm\") pod \"maas-api-revoke-sub-31b20f9f-bbs4g\" (UID: \"2806e408-4be8-4af6-90b3-4c122a06b402\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31b20f9f-bbs4g" Nov 03 21:01:18.903172 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:18.903132 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:01:18.905087 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:18.905055 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77vtd\"" Nov 03 21:01:18.967828 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:18.967006 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2806e408-4be8-4af6-90b3-4c122a06b402-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31b20f9f-bbs4g\" (UID: \"2806e408-4be8-4af6-90b3-4c122a06b402\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31b20f9f-bbs4g" Nov 03 21:01:18.967828 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:18.967204 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvnsm\" (UniqueName: \"kubernetes.io/projected/2806e408-4be8-4af6-90b3-4c122a06b402-kube-api-access-lvnsm\") pod \"maas-api-revoke-sub-31b20f9f-bbs4g\" (UID: \"2806e408-4be8-4af6-90b3-4c122a06b402\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31b20f9f-bbs4g" Nov 03 21:01:18.967828 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:18.967510 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2806e408-4be8-4af6-90b3-4c122a06b402-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31b20f9f-bbs4g\" (UID: \"2806e408-4be8-4af6-90b3-4c122a06b402\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31b20f9f-bbs4g" Nov 03 21:01:18.969245 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:18.969218 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2806e408-4be8-4af6-90b3-4c122a06b402-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31b20f9f-bbs4g\" (UID: \"2806e408-4be8-4af6-90b3-4c122a06b402\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31b20f9f-bbs4g" Nov 03 21:01:18.970496 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:18.970476 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2806e408-4be8-4af6-90b3-4c122a06b402-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31b20f9f-bbs4g\" (UID: \"2806e408-4be8-4af6-90b3-4c122a06b402\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31b20f9f-bbs4g" Nov 03 21:01:19.031832 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:19.031796 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvnsm\" (UniqueName: \"kubernetes.io/projected/2806e408-4be8-4af6-90b3-4c122a06b402-kube-api-access-lvnsm\") pod \"maas-api-revoke-sub-31b20f9f-bbs4g\" (UID: \"2806e408-4be8-4af6-90b3-4c122a06b402\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31b20f9f-bbs4g" Nov 03 21:01:19.120892 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:19.120799 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31b20f9f-bbs4g" Nov 03 21:01:19.511697 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:01:19.511665 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2806e408_4be8_4af6_90b3_4c122a06b402.slice/crio-4dc644d7f5a54bb1048f24c2a278a103c8f111c086ac1083f7d3489981e97238 WatchSource:0}: Error finding container 4dc644d7f5a54bb1048f24c2a278a103c8f111c086ac1083f7d3489981e97238: Status 404 returned error can't find the container with id 4dc644d7f5a54bb1048f24c2a278a103c8f111c086ac1083f7d3489981e97238 Nov 03 21:01:19.512183 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:19.512162 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31b20f9f-bbs4g"] Nov 03 21:01:19.980393 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:19.980356 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31b20f9f-bbs4g" event={"ID":"2806e408-4be8-4af6-90b3-4c122a06b402","Type":"ContainerStarted","Data":"b92cbc038d5b3ce38786fc7f9871117629f4454fde3c707dfc65d4f4a6386004"} Nov 03 21:01:19.980875 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:19.980419 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31b20f9f-bbs4g" event={"ID":"2806e408-4be8-4af6-90b3-4c122a06b402","Type":"ContainerStarted","Data":"4dc644d7f5a54bb1048f24c2a278a103c8f111c086ac1083f7d3489981e97238"} Nov 03 21:01:20.099897 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:20.099792 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31b20f9f-bbs4g" podStartSLOduration=2.099778142 podStartE2EDuration="2.099778142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:01:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:01:20.09466805 +0000 UTC m=+1315.319987894" watchObservedRunningTime="2026-11-03 21:01:20.099778142 +0000 UTC m=+1315.325097930" Nov 03 21:01:20.994544 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:20.994510 4305 generic.go:356] "Generic (PLEG): container finished" podID="2806e408-4be8-4af6-90b3-4c122a06b402" containerID="b92cbc038d5b3ce38786fc7f9871117629f4454fde3c707dfc65d4f4a6386004" exitCode=0 Nov 03 21:01:20.995364 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:20.994629 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31b20f9f-bbs4g" event={"ID":"2806e408-4be8-4af6-90b3-4c122a06b402","Type":"ContainerDied","Data":"b92cbc038d5b3ce38786fc7f9871117629f4454fde3c707dfc65d4f4a6386004"} Nov 03 21:01:21.389244 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:21.389121 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx"] Nov 03 21:01:21.393814 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:21.393789 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx" Nov 03 21:01:21.503003 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:21.502966 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx"] Nov 03 21:01:21.509726 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:21.509684 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36f2d843-7d6a-4606-b6b3-6904a2b6cc29-istio-data\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx\" (UID: \"36f2d843-7d6a-4606-b6b3-6904a2b6cc29\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx" Nov 03 21:01:21.509921 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:21.509741 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36f2d843-7d6a-4606-b6b3-6904a2b6cc29-istio-token\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx\" (UID: \"36f2d843-7d6a-4606-b6b3-6904a2b6cc29\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx" Nov 03 21:01:21.509921 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:21.509855 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36f2d843-7d6a-4606-b6b3-6904a2b6cc29-istio-podinfo\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx\" (UID: \"36f2d843-7d6a-4606-b6b3-6904a2b6cc29\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx" Nov 03 21:01:21.509921 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:21.509888 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36f2d843-7d6a-4606-b6b3-6904a2b6cc29-credential-socket\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx\" (UID: \"36f2d843-7d6a-4606-b6b3-6904a2b6cc29\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx" Nov 03 21:01:21.510060 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:21.510001 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36f2d843-7d6a-4606-b6b3-6904a2b6cc29-istiod-ca-cert\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx\" (UID: \"36f2d843-7d6a-4606-b6b3-6904a2b6cc29\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx" Nov 03 21:01:21.510060 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:21.510040 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36f2d843-7d6a-4606-b6b3-6904a2b6cc29-workload-socket\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx\" (UID: \"36f2d843-7d6a-4606-b6b3-6904a2b6cc29\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx" Nov 03 21:01:21.510219 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:21.510200 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36f2d843-7d6a-4606-b6b3-6904a2b6cc29-workload-certs\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx\" (UID: \"36f2d843-7d6a-4606-b6b3-6904a2b6cc29\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx" Nov 03 21:01:21.510255 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:21.510240 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36f2d843-7d6a-4606-b6b3-6904a2b6cc29-istio-envoy\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx\" (UID: \"36f2d843-7d6a-4606-b6b3-6904a2b6cc29\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx" Nov 03 21:01:21.510327 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:21.510315 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tkw7\" (UniqueName: \"kubernetes.io/projected/36f2d843-7d6a-4606-b6b3-6904a2b6cc29-kube-api-access-2tkw7\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx\" (UID: \"36f2d843-7d6a-4606-b6b3-6904a2b6cc29\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx" Nov 03 21:01:21.611739 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:21.611692 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36f2d843-7d6a-4606-b6b3-6904a2b6cc29-istio-podinfo\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx\" (UID: \"36f2d843-7d6a-4606-b6b3-6904a2b6cc29\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx" Nov 03 21:01:21.611739 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:21.611740 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36f2d843-7d6a-4606-b6b3-6904a2b6cc29-credential-socket\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx\" (UID: \"36f2d843-7d6a-4606-b6b3-6904a2b6cc29\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx" Nov 03 21:01:21.612017 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:21.611788 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36f2d843-7d6a-4606-b6b3-6904a2b6cc29-istiod-ca-cert\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx\" (UID: \"36f2d843-7d6a-4606-b6b3-6904a2b6cc29\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx" Nov 03 21:01:21.612017 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:21.611816 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36f2d843-7d6a-4606-b6b3-6904a2b6cc29-workload-socket\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx\" (UID: \"36f2d843-7d6a-4606-b6b3-6904a2b6cc29\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx" Nov 03 21:01:21.612017 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:21.611933 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36f2d843-7d6a-4606-b6b3-6904a2b6cc29-workload-certs\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx\" (UID: \"36f2d843-7d6a-4606-b6b3-6904a2b6cc29\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx" Nov 03 21:01:21.612017 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:21.611958 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36f2d843-7d6a-4606-b6b3-6904a2b6cc29-istio-envoy\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx\" (UID: \"36f2d843-7d6a-4606-b6b3-6904a2b6cc29\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx" Nov 03 21:01:21.612222 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:21.612033 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tkw7\" (UniqueName: \"kubernetes.io/projected/36f2d843-7d6a-4606-b6b3-6904a2b6cc29-kube-api-access-2tkw7\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx\" (UID: \"36f2d843-7d6a-4606-b6b3-6904a2b6cc29\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx" Nov 03 21:01:21.612222 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:21.612208 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36f2d843-7d6a-4606-b6b3-6904a2b6cc29-istio-data\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx\" (UID: \"36f2d843-7d6a-4606-b6b3-6904a2b6cc29\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx" Nov 03 21:01:21.612385 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:21.612243 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36f2d843-7d6a-4606-b6b3-6904a2b6cc29-istio-token\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx\" (UID: \"36f2d843-7d6a-4606-b6b3-6904a2b6cc29\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx" Nov 03 21:01:21.612385 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:21.612352 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36f2d843-7d6a-4606-b6b3-6904a2b6cc29-credential-socket\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx\" (UID: \"36f2d843-7d6a-4606-b6b3-6904a2b6cc29\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx" Nov 03 21:01:21.612811 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:21.612764 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36f2d843-7d6a-4606-b6b3-6904a2b6cc29-workload-certs\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx\" (UID: \"36f2d843-7d6a-4606-b6b3-6904a2b6cc29\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx" Nov 03 21:01:21.612811 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:21.612720 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36f2d843-7d6a-4606-b6b3-6904a2b6cc29-workload-socket\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx\" (UID: \"36f2d843-7d6a-4606-b6b3-6904a2b6cc29\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx" Nov 03 21:01:21.612811 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:21.612805 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36f2d843-7d6a-4606-b6b3-6904a2b6cc29-istio-data\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx\" (UID: \"36f2d843-7d6a-4606-b6b3-6904a2b6cc29\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx" Nov 03 21:01:21.612961 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:21.612864 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36f2d843-7d6a-4606-b6b3-6904a2b6cc29-istiod-ca-cert\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx\" (UID: \"36f2d843-7d6a-4606-b6b3-6904a2b6cc29\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx" Nov 03 21:01:21.614539 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:21.614521 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36f2d843-7d6a-4606-b6b3-6904a2b6cc29-istio-envoy\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx\" (UID: \"36f2d843-7d6a-4606-b6b3-6904a2b6cc29\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx" Nov 03 21:01:21.614695 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:21.614676 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36f2d843-7d6a-4606-b6b3-6904a2b6cc29-istio-podinfo\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx\" (UID: \"36f2d843-7d6a-4606-b6b3-6904a2b6cc29\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx" Nov 03 21:01:21.674665 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:21.674523 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tkw7\" (UniqueName: \"kubernetes.io/projected/36f2d843-7d6a-4606-b6b3-6904a2b6cc29-kube-api-access-2tkw7\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx\" (UID: \"36f2d843-7d6a-4606-b6b3-6904a2b6cc29\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx" Nov 03 21:01:21.676750 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:21.676716 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36f2d843-7d6a-4606-b6b3-6904a2b6cc29-istio-token\") pod \"e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx\" (UID: \"36f2d843-7d6a-4606-b6b3-6904a2b6cc29\") " pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx" Nov 03 21:01:21.708700 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:21.708667 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx" Nov 03 21:01:22.158145 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.158104 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx"] Nov 03 21:01:22.160709 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:01:22.160673 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36f2d843_7d6a_4606_b6b3_6904a2b6cc29.slice/crio-67144fb97d0431f7c086651fcdb49927a0bae010aebbd211c9419b7c124021fc WatchSource:0}: Error finding container 67144fb97d0431f7c086651fcdb49927a0bae010aebbd211c9419b7c124021fc: Status 404 returned error can't find the container with id 67144fb97d0431f7c086651fcdb49927a0bae010aebbd211c9419b7c124021fc Nov 03 21:01:22.164015 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.163985 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v"] Nov 03 21:01:22.167003 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.166968 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:01:22.167112 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.167040 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:01:22.167112 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.167072 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:01:22.171814 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.171793 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v" Nov 03 21:01:22.176083 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.176057 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31b20f9f-bbs4g" Nov 03 21:01:22.205484 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.205449 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v"] Nov 03 21:01:22.206172 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.206158 4305 cpu_manager.go:414] "containerMap: removing container" podUID="2806e408-4be8-4af6-90b3-4c122a06b402" containerName="revoke-keys" Nov 03 21:01:22.206221 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.206174 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2806e408-4be8-4af6-90b3-4c122a06b402" containerName="revoke-keys" Nov 03 21:01:22.206289 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.206280 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="2806e408-4be8-4af6-90b3-4c122a06b402" containerName="revoke-keys" Nov 03 21:01:22.210521 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.210491 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v" Nov 03 21:01:22.241069 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.241041 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2806e408-4be8-4af6-90b3-4c122a06b402-kube-api-access-lvnsm\" (UniqueName: \"kubernetes.io/projected/2806e408-4be8-4af6-90b3-4c122a06b402-kube-api-access-lvnsm\") pod \"2806e408-4be8-4af6-90b3-4c122a06b402\" (UID: \"2806e408-4be8-4af6-90b3-4c122a06b402\") " Nov 03 21:01:22.241403 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.241234 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2806e408-4be8-4af6-90b3-4c122a06b402-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2806e408-4be8-4af6-90b3-4c122a06b402-maas-api-openshift-ca\") pod \"2806e408-4be8-4af6-90b3-4c122a06b402\" (UID: \"2806e408-4be8-4af6-90b3-4c122a06b402\") " Nov 03 21:01:22.241572 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.241452 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2806e408-4be8-4af6-90b3-4c122a06b402-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2806e408-4be8-4af6-90b3-4c122a06b402-maas-api-xks-ca\") pod \"2806e408-4be8-4af6-90b3-4c122a06b402\" (UID: \"2806e408-4be8-4af6-90b3-4c122a06b402\") " Nov 03 21:01:22.242181 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.241756 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24ff37e-1709-4f11-b0e8-9302b9f3d9ad-credential-socket\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v\" (UID: \"b24ff37e-1709-4f11-b0e8-9302b9f3d9ad\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v" Nov 03 21:01:22.242181 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.241809 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c86md\" (UniqueName: \"kubernetes.io/projected/b24ff37e-1709-4f11-b0e8-9302b9f3d9ad-kube-api-access-c86md\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v\" (UID: \"b24ff37e-1709-4f11-b0e8-9302b9f3d9ad\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v" Nov 03 21:01:22.242181 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.241937 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b24ff37e-1709-4f11-b0e8-9302b9f3d9ad-istio-envoy\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v\" (UID: \"b24ff37e-1709-4f11-b0e8-9302b9f3d9ad\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v" Nov 03 21:01:22.242181 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.242021 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24ff37e-1709-4f11-b0e8-9302b9f3d9ad-workload-socket\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v\" (UID: \"b24ff37e-1709-4f11-b0e8-9302b9f3d9ad\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v" Nov 03 21:01:22.242181 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.242048 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2806e408-4be8-4af6-90b3-4c122a06b402-maas-api-openshift-ca" pod "2806e408-4be8-4af6-90b3-4c122a06b402" (UID: "2806e408-4be8-4af6-90b3-4c122a06b402"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:01:22.242181 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.242138 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b24ff37e-1709-4f11-b0e8-9302b9f3d9ad-istio-token\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v\" (UID: \"b24ff37e-1709-4f11-b0e8-9302b9f3d9ad\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v" Nov 03 21:01:22.242507 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.242297 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b24ff37e-1709-4f11-b0e8-9302b9f3d9ad-istio-podinfo\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v\" (UID: \"b24ff37e-1709-4f11-b0e8-9302b9f3d9ad\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v" Nov 03 21:01:22.242507 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.242327 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b24ff37e-1709-4f11-b0e8-9302b9f3d9ad-istiod-ca-cert\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v\" (UID: \"b24ff37e-1709-4f11-b0e8-9302b9f3d9ad\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v" Nov 03 21:01:22.242507 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.242413 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b24ff37e-1709-4f11-b0e8-9302b9f3d9ad-istio-data\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v\" (UID: \"b24ff37e-1709-4f11-b0e8-9302b9f3d9ad\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v" Nov 03 21:01:22.242618 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.242544 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b24ff37e-1709-4f11-b0e8-9302b9f3d9ad-workload-certs\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v\" (UID: \"b24ff37e-1709-4f11-b0e8-9302b9f3d9ad\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v" Nov 03 21:01:22.242672 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.242655 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2806e408-4be8-4af6-90b3-4c122a06b402-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:01:22.243447 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.243418 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2806e408-4be8-4af6-90b3-4c122a06b402-kube-api-access-lvnsm" pod "2806e408-4be8-4af6-90b3-4c122a06b402" (UID: "2806e408-4be8-4af6-90b3-4c122a06b402"). InnerVolumeSpecName "kube-api-access-lvnsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:01:22.243526 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.243487 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2806e408-4be8-4af6-90b3-4c122a06b402-maas-api-xks-ca" pod "2806e408-4be8-4af6-90b3-4c122a06b402" (UID: "2806e408-4be8-4af6-90b3-4c122a06b402"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:01:22.255972 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.255886 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v"] Nov 03 21:01:22.316460 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.315582 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v"] Nov 03 21:01:22.343578 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.343543 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b24ff37e-1709-4f11-b0e8-9302b9f3d9ad-istio-envoy\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v\" (UID: \"b24ff37e-1709-4f11-b0e8-9302b9f3d9ad\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v" Nov 03 21:01:22.343984 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.343961 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26613665-1067-4ba1-9b16-f84e8c583c2a-istio-data\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v\" (UID: \"26613665-1067-4ba1-9b16-f84e8c583c2a\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v" Nov 03 21:01:22.344108 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.344045 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t2qs\" (UniqueName: \"kubernetes.io/projected/26613665-1067-4ba1-9b16-f84e8c583c2a-kube-api-access-4t2qs\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v\" (UID: \"26613665-1067-4ba1-9b16-f84e8c583c2a\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v" Nov 03 21:01:22.344108 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.344074 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24ff37e-1709-4f11-b0e8-9302b9f3d9ad-workload-socket\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v\" (UID: \"b24ff37e-1709-4f11-b0e8-9302b9f3d9ad\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v" Nov 03 21:01:22.344223 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.344138 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26613665-1067-4ba1-9b16-f84e8c583c2a-workload-socket\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v\" (UID: \"26613665-1067-4ba1-9b16-f84e8c583c2a\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v" Nov 03 21:01:22.344391 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.344375 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24ff37e-1709-4f11-b0e8-9302b9f3d9ad-workload-socket\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v\" (UID: \"b24ff37e-1709-4f11-b0e8-9302b9f3d9ad\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v" Nov 03 21:01:22.344501 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.344485 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b24ff37e-1709-4f11-b0e8-9302b9f3d9ad-istio-token\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v\" (UID: \"b24ff37e-1709-4f11-b0e8-9302b9f3d9ad\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v" Nov 03 21:01:22.344621 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.344602 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26613665-1067-4ba1-9b16-f84e8c583c2a-istiod-ca-cert\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v\" (UID: \"26613665-1067-4ba1-9b16-f84e8c583c2a\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v" Nov 03 21:01:22.344684 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.344667 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b24ff37e-1709-4f11-b0e8-9302b9f3d9ad-istio-podinfo\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v\" (UID: \"b24ff37e-1709-4f11-b0e8-9302b9f3d9ad\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v" Nov 03 21:01:22.344738 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.344696 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b24ff37e-1709-4f11-b0e8-9302b9f3d9ad-istiod-ca-cert\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v\" (UID: \"b24ff37e-1709-4f11-b0e8-9302b9f3d9ad\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v" Nov 03 21:01:22.344794 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.344761 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26613665-1067-4ba1-9b16-f84e8c583c2a-istio-token\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v\" (UID: \"26613665-1067-4ba1-9b16-f84e8c583c2a\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v" Nov 03 21:01:22.344854 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.344805 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b24ff37e-1709-4f11-b0e8-9302b9f3d9ad-istio-data\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v\" (UID: \"b24ff37e-1709-4f11-b0e8-9302b9f3d9ad\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v" Nov 03 21:01:22.344854 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.344848 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26613665-1067-4ba1-9b16-f84e8c583c2a-istio-podinfo\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v\" (UID: \"26613665-1067-4ba1-9b16-f84e8c583c2a\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v" Nov 03 21:01:22.344961 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.344888 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26613665-1067-4ba1-9b16-f84e8c583c2a-workload-certs\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v\" (UID: \"26613665-1067-4ba1-9b16-f84e8c583c2a\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v" Nov 03 21:01:22.344961 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.344942 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b24ff37e-1709-4f11-b0e8-9302b9f3d9ad-workload-certs\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v\" (UID: \"b24ff37e-1709-4f11-b0e8-9302b9f3d9ad\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v" Nov 03 21:01:22.345076 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.345005 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26613665-1067-4ba1-9b16-f84e8c583c2a-istio-envoy\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v\" (UID: \"26613665-1067-4ba1-9b16-f84e8c583c2a\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v" Nov 03 21:01:22.345076 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.345053 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24ff37e-1709-4f11-b0e8-9302b9f3d9ad-credential-socket\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v\" (UID: \"b24ff37e-1709-4f11-b0e8-9302b9f3d9ad\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v" Nov 03 21:01:22.345353 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.345326 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b24ff37e-1709-4f11-b0e8-9302b9f3d9ad-istio-data\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v\" (UID: \"b24ff37e-1709-4f11-b0e8-9302b9f3d9ad\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v" Nov 03 21:01:22.345630 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.345609 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24ff37e-1709-4f11-b0e8-9302b9f3d9ad-credential-socket\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v\" (UID: \"b24ff37e-1709-4f11-b0e8-9302b9f3d9ad\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v" Nov 03 21:01:22.345733 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.345715 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b24ff37e-1709-4f11-b0e8-9302b9f3d9ad-istiod-ca-cert\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v\" (UID: \"b24ff37e-1709-4f11-b0e8-9302b9f3d9ad\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v" Nov 03 21:01:22.346621 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.345993 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c86md\" (UniqueName: \"kubernetes.io/projected/b24ff37e-1709-4f11-b0e8-9302b9f3d9ad-kube-api-access-c86md\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v\" (UID: \"b24ff37e-1709-4f11-b0e8-9302b9f3d9ad\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v" Nov 03 21:01:22.346726 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.346665 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26613665-1067-4ba1-9b16-f84e8c583c2a-credential-socket\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v\" (UID: \"26613665-1067-4ba1-9b16-f84e8c583c2a\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v" Nov 03 21:01:22.346726 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.346125 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b24ff37e-1709-4f11-b0e8-9302b9f3d9ad-workload-certs\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v\" (UID: \"b24ff37e-1709-4f11-b0e8-9302b9f3d9ad\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v" Nov 03 21:01:22.346837 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.346809 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2806e408-4be8-4af6-90b3-4c122a06b402-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:01:22.346837 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.346823 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvnsm\" (UniqueName: \"kubernetes.io/projected/2806e408-4be8-4af6-90b3-4c122a06b402-kube-api-access-lvnsm\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:01:22.348712 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.348688 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b24ff37e-1709-4f11-b0e8-9302b9f3d9ad-istio-envoy\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v\" (UID: \"b24ff37e-1709-4f11-b0e8-9302b9f3d9ad\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v" Nov 03 21:01:22.349530 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.349508 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b24ff37e-1709-4f11-b0e8-9302b9f3d9ad-istio-podinfo\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v\" (UID: \"b24ff37e-1709-4f11-b0e8-9302b9f3d9ad\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v" Nov 03 21:01:22.377155 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.377122 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44"] Nov 03 21:01:22.383013 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.382984 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44" Nov 03 21:01:22.385083 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.385056 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ffa5bc4-2lq5j"] Nov 03 21:01:22.390561 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.390529 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ffa5bc4-2lq5j" Nov 03 21:01:22.405260 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.405197 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b24ff37e-1709-4f11-b0e8-9302b9f3d9ad-istio-token\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v\" (UID: \"b24ff37e-1709-4f11-b0e8-9302b9f3d9ad\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v" Nov 03 21:01:22.409218 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.408947 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c86md\" (UniqueName: \"kubernetes.io/projected/b24ff37e-1709-4f11-b0e8-9302b9f3d9ad-kube-api-access-c86md\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v\" (UID: \"b24ff37e-1709-4f11-b0e8-9302b9f3d9ad\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v" Nov 03 21:01:22.444522 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.443720 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44"] Nov 03 21:01:22.448803 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.448603 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4b8z\" (UniqueName: \"kubernetes.io/projected/4dbbf182-5d1d-4a54-8f71-bb58afa654e1-kube-api-access-x4b8z\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44\" (UID: \"4dbbf182-5d1d-4a54-8f71-bb58afa654e1\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44" Nov 03 21:01:22.448803 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.448662 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dbbf182-5d1d-4a54-8f71-bb58afa654e1-workload-socket\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44\" (UID: \"4dbbf182-5d1d-4a54-8f71-bb58afa654e1\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44" Nov 03 21:01:22.449193 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.449160 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26613665-1067-4ba1-9b16-f84e8c583c2a-istio-envoy\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v\" (UID: \"26613665-1067-4ba1-9b16-f84e8c583c2a\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v" Nov 03 21:01:22.449314 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.449214 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dbbf182-5d1d-4a54-8f71-bb58afa654e1-istio-data\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44\" (UID: \"4dbbf182-5d1d-4a54-8f71-bb58afa654e1\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44" Nov 03 21:01:22.449314 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.449242 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce39a4a7-549b-4163-aab1-ac6581c9c451-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ffa5bc4-2lq5j\" (UID: \"ce39a4a7-549b-4163-aab1-ac6581c9c451\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ffa5bc4-2lq5j" Nov 03 21:01:22.449314 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.449298 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwtx9\" (UniqueName: \"kubernetes.io/projected/ce39a4a7-549b-4163-aab1-ac6581c9c451-kube-api-access-cwtx9\") pod \"maas-api-revoke-sub-6ffa5bc4-2lq5j\" (UID: \"ce39a4a7-549b-4163-aab1-ac6581c9c451\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ffa5bc4-2lq5j" Nov 03 21:01:22.449539 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.449340 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26613665-1067-4ba1-9b16-f84e8c583c2a-credential-socket\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v\" (UID: \"26613665-1067-4ba1-9b16-f84e8c583c2a\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v" Nov 03 21:01:22.449539 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.449419 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26613665-1067-4ba1-9b16-f84e8c583c2a-istio-data\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v\" (UID: \"26613665-1067-4ba1-9b16-f84e8c583c2a\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v" Nov 03 21:01:22.449539 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.449465 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t2qs\" (UniqueName: \"kubernetes.io/projected/26613665-1067-4ba1-9b16-f84e8c583c2a-kube-api-access-4t2qs\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v\" (UID: \"26613665-1067-4ba1-9b16-f84e8c583c2a\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v" Nov 03 21:01:22.449539 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.449500 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dbbf182-5d1d-4a54-8f71-bb58afa654e1-istiod-ca-cert\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44\" (UID: \"4dbbf182-5d1d-4a54-8f71-bb58afa654e1\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44" Nov 03 21:01:22.449750 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.449550 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce39a4a7-549b-4163-aab1-ac6581c9c451-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ffa5bc4-2lq5j\" (UID: \"ce39a4a7-549b-4163-aab1-ac6581c9c451\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ffa5bc4-2lq5j" Nov 03 21:01:22.449750 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.449586 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dbbf182-5d1d-4a54-8f71-bb58afa654e1-credential-socket\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44\" (UID: \"4dbbf182-5d1d-4a54-8f71-bb58afa654e1\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44" Nov 03 21:01:22.449750 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.449616 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26613665-1067-4ba1-9b16-f84e8c583c2a-workload-socket\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v\" (UID: \"26613665-1067-4ba1-9b16-f84e8c583c2a\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v" Nov 03 21:01:22.449750 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.449646 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dbbf182-5d1d-4a54-8f71-bb58afa654e1-istio-token\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44\" (UID: \"4dbbf182-5d1d-4a54-8f71-bb58afa654e1\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44" Nov 03 21:01:22.449960 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.449765 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dbbf182-5d1d-4a54-8f71-bb58afa654e1-istio-envoy\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44\" (UID: \"4dbbf182-5d1d-4a54-8f71-bb58afa654e1\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44" Nov 03 21:01:22.449960 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.449813 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26613665-1067-4ba1-9b16-f84e8c583c2a-istiod-ca-cert\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v\" (UID: \"26613665-1067-4ba1-9b16-f84e8c583c2a\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v" Nov 03 21:01:22.449960 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.449934 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26613665-1067-4ba1-9b16-f84e8c583c2a-istio-token\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v\" (UID: \"26613665-1067-4ba1-9b16-f84e8c583c2a\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v" Nov 03 21:01:22.450115 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.450005 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26613665-1067-4ba1-9b16-f84e8c583c2a-istio-podinfo\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v\" (UID: \"26613665-1067-4ba1-9b16-f84e8c583c2a\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v" Nov 03 21:01:22.450115 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.450031 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dbbf182-5d1d-4a54-8f71-bb58afa654e1-workload-certs\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44\" (UID: \"4dbbf182-5d1d-4a54-8f71-bb58afa654e1\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44" Nov 03 21:01:22.450115 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.450063 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dbbf182-5d1d-4a54-8f71-bb58afa654e1-istio-podinfo\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44\" (UID: \"4dbbf182-5d1d-4a54-8f71-bb58afa654e1\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44" Nov 03 21:01:22.450115 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.450099 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26613665-1067-4ba1-9b16-f84e8c583c2a-workload-certs\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v\" (UID: \"26613665-1067-4ba1-9b16-f84e8c583c2a\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v" Nov 03 21:01:22.450870 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.450480 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26613665-1067-4ba1-9b16-f84e8c583c2a-workload-socket\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v\" (UID: \"26613665-1067-4ba1-9b16-f84e8c583c2a\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v" Nov 03 21:01:22.451474 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.451414 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26613665-1067-4ba1-9b16-f84e8c583c2a-istiod-ca-cert\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v\" (UID: \"26613665-1067-4ba1-9b16-f84e8c583c2a\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v" Nov 03 21:01:22.451997 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.451960 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26613665-1067-4ba1-9b16-f84e8c583c2a-credential-socket\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v\" (UID: \"26613665-1067-4ba1-9b16-f84e8c583c2a\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v" Nov 03 21:01:22.452185 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.452164 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26613665-1067-4ba1-9b16-f84e8c583c2a-workload-certs\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v\" (UID: \"26613665-1067-4ba1-9b16-f84e8c583c2a\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v" Nov 03 21:01:22.452285 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.452273 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26613665-1067-4ba1-9b16-f84e8c583c2a-istio-data\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v\" (UID: \"26613665-1067-4ba1-9b16-f84e8c583c2a\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v" Nov 03 21:01:22.455600 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.455574 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26613665-1067-4ba1-9b16-f84e8c583c2a-istio-podinfo\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v\" (UID: \"26613665-1067-4ba1-9b16-f84e8c583c2a\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v" Nov 03 21:01:22.456023 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.456000 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26613665-1067-4ba1-9b16-f84e8c583c2a-istio-envoy\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v\" (UID: \"26613665-1067-4ba1-9b16-f84e8c583c2a\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v" Nov 03 21:01:22.458702 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.458643 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ffa5bc4-2lq5j"] Nov 03 21:01:22.490403 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.490356 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v" Nov 03 21:01:22.510967 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.510886 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26613665-1067-4ba1-9b16-f84e8c583c2a-istio-token\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v\" (UID: \"26613665-1067-4ba1-9b16-f84e8c583c2a\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v" Nov 03 21:01:22.510967 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.510930 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t2qs\" (UniqueName: \"kubernetes.io/projected/26613665-1067-4ba1-9b16-f84e8c583c2a-kube-api-access-4t2qs\") pod \"e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v\" (UID: \"26613665-1067-4ba1-9b16-f84e8c583c2a\") " pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v" Nov 03 21:01:22.536250 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.535986 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v" Nov 03 21:01:22.554306 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.553864 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwtx9\" (UniqueName: \"kubernetes.io/projected/ce39a4a7-549b-4163-aab1-ac6581c9c451-kube-api-access-cwtx9\") pod \"maas-api-revoke-sub-6ffa5bc4-2lq5j\" (UID: \"ce39a4a7-549b-4163-aab1-ac6581c9c451\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ffa5bc4-2lq5j" Nov 03 21:01:22.554712 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.554689 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dbbf182-5d1d-4a54-8f71-bb58afa654e1-istiod-ca-cert\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44\" (UID: \"4dbbf182-5d1d-4a54-8f71-bb58afa654e1\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44" Nov 03 21:01:22.555638 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.555531 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dbbf182-5d1d-4a54-8f71-bb58afa654e1-istiod-ca-cert\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44\" (UID: \"4dbbf182-5d1d-4a54-8f71-bb58afa654e1\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44" Nov 03 21:01:22.555729 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.555669 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce39a4a7-549b-4163-aab1-ac6581c9c451-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ffa5bc4-2lq5j\" (UID: \"ce39a4a7-549b-4163-aab1-ac6581c9c451\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ffa5bc4-2lq5j" Nov 03 21:01:22.556392 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.556367 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce39a4a7-549b-4163-aab1-ac6581c9c451-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ffa5bc4-2lq5j\" (UID: \"ce39a4a7-549b-4163-aab1-ac6581c9c451\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ffa5bc4-2lq5j" Nov 03 21:01:22.556760 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.556736 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dbbf182-5d1d-4a54-8f71-bb58afa654e1-credential-socket\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44\" (UID: \"4dbbf182-5d1d-4a54-8f71-bb58afa654e1\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44" Nov 03 21:01:22.556842 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.555704 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dbbf182-5d1d-4a54-8f71-bb58afa654e1-credential-socket\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44\" (UID: \"4dbbf182-5d1d-4a54-8f71-bb58afa654e1\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44" Nov 03 21:01:22.556842 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.556833 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dbbf182-5d1d-4a54-8f71-bb58afa654e1-istio-token\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44\" (UID: \"4dbbf182-5d1d-4a54-8f71-bb58afa654e1\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44" Nov 03 21:01:22.557244 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.557221 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dbbf182-5d1d-4a54-8f71-bb58afa654e1-istio-envoy\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44\" (UID: \"4dbbf182-5d1d-4a54-8f71-bb58afa654e1\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44" Nov 03 21:01:22.557794 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.557772 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dbbf182-5d1d-4a54-8f71-bb58afa654e1-workload-certs\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44\" (UID: \"4dbbf182-5d1d-4a54-8f71-bb58afa654e1\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44" Nov 03 21:01:22.558150 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.558096 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dbbf182-5d1d-4a54-8f71-bb58afa654e1-workload-certs\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44\" (UID: \"4dbbf182-5d1d-4a54-8f71-bb58afa654e1\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44" Nov 03 21:01:22.558303 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.558285 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dbbf182-5d1d-4a54-8f71-bb58afa654e1-istio-podinfo\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44\" (UID: \"4dbbf182-5d1d-4a54-8f71-bb58afa654e1\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44" Nov 03 21:01:22.558479 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.558460 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4b8z\" (UniqueName: \"kubernetes.io/projected/4dbbf182-5d1d-4a54-8f71-bb58afa654e1-kube-api-access-x4b8z\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44\" (UID: \"4dbbf182-5d1d-4a54-8f71-bb58afa654e1\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44" Nov 03 21:01:22.558605 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.558590 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dbbf182-5d1d-4a54-8f71-bb58afa654e1-workload-socket\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44\" (UID: \"4dbbf182-5d1d-4a54-8f71-bb58afa654e1\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44" Nov 03 21:01:22.558767 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.558753 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dbbf182-5d1d-4a54-8f71-bb58afa654e1-istio-data\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44\" (UID: \"4dbbf182-5d1d-4a54-8f71-bb58afa654e1\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44" Nov 03 21:01:22.558879 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.558865 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce39a4a7-549b-4163-aab1-ac6581c9c451-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ffa5bc4-2lq5j\" (UID: \"ce39a4a7-549b-4163-aab1-ac6581c9c451\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ffa5bc4-2lq5j" Nov 03 21:01:22.559817 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.559465 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dbbf182-5d1d-4a54-8f71-bb58afa654e1-workload-socket\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44\" (UID: \"4dbbf182-5d1d-4a54-8f71-bb58afa654e1\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44" Nov 03 21:01:22.559817 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.559720 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dbbf182-5d1d-4a54-8f71-bb58afa654e1-istio-data\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44\" (UID: \"4dbbf182-5d1d-4a54-8f71-bb58afa654e1\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44" Nov 03 21:01:22.560657 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.560626 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dbbf182-5d1d-4a54-8f71-bb58afa654e1-istio-envoy\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44\" (UID: \"4dbbf182-5d1d-4a54-8f71-bb58afa654e1\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44" Nov 03 21:01:22.564742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.561906 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce39a4a7-549b-4163-aab1-ac6581c9c451-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ffa5bc4-2lq5j\" (UID: \"ce39a4a7-549b-4163-aab1-ac6581c9c451\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ffa5bc4-2lq5j" Nov 03 21:01:22.564742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.562090 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dbbf182-5d1d-4a54-8f71-bb58afa654e1-istio-podinfo\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44\" (UID: \"4dbbf182-5d1d-4a54-8f71-bb58afa654e1\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44" Nov 03 21:01:22.622336 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.621613 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwtx9\" (UniqueName: \"kubernetes.io/projected/ce39a4a7-549b-4163-aab1-ac6581c9c451-kube-api-access-cwtx9\") pod \"maas-api-revoke-sub-6ffa5bc4-2lq5j\" (UID: \"ce39a4a7-549b-4163-aab1-ac6581c9c451\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ffa5bc4-2lq5j" Nov 03 21:01:22.623316 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.623264 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4b8z\" (UniqueName: \"kubernetes.io/projected/4dbbf182-5d1d-4a54-8f71-bb58afa654e1-kube-api-access-x4b8z\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44\" (UID: \"4dbbf182-5d1d-4a54-8f71-bb58afa654e1\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44" Nov 03 21:01:22.623978 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.623938 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dbbf182-5d1d-4a54-8f71-bb58afa654e1-istio-token\") pod \"e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44\" (UID: \"4dbbf182-5d1d-4a54-8f71-bb58afa654e1\") " pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44" Nov 03 21:01:22.709225 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.709186 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44" Nov 03 21:01:22.728386 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.724710 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ffa5bc4-2lq5j" Nov 03 21:01:22.909635 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.909427 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v"] Nov 03 21:01:22.909635 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.909484 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:01:22.909635 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.909564 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:01:22.909635 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.909613 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:01:22.966736 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:01:22.966689 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26613665_1067_4ba1_9b16_f84e8c583c2a.slice/crio-7a8de5f3446cbc61d84b9953de793ee6907da868f57f6cf4d763c54b45239cd1 WatchSource:0}: Error finding container 7a8de5f3446cbc61d84b9953de793ee6907da868f57f6cf4d763c54b45239cd1: Status 404 returned error can't find the container with id 7a8de5f3446cbc61d84b9953de793ee6907da868f57f6cf4d763c54b45239cd1 Nov 03 21:01:22.968319 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.968294 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v"] Nov 03 21:01:22.969633 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.969596 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:01:22.969716 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.969676 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:01:22.969755 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:22.969720 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:01:23.034364 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:23.034332 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v" event={"ID":"b24ff37e-1709-4f11-b0e8-9302b9f3d9ad","Type":"ContainerStarted","Data":"6e1aae7afc7ed5b987a6f7b40e8fb7e76b95e6fd2665fa3f61350f63239d6d27"} Nov 03 21:01:23.034602 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:23.034584 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v" event={"ID":"b24ff37e-1709-4f11-b0e8-9302b9f3d9ad","Type":"ContainerStarted","Data":"57dfd54234ef7a9ff5f0ffc62dec91d4d88c544721ed16c18400c0bc17fc7b78"} Nov 03 21:01:23.038982 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:23.038896 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx" event={"ID":"36f2d843-7d6a-4606-b6b3-6904a2b6cc29","Type":"ContainerStarted","Data":"8818f76d7f9e22d97b1005dad732e19ae4011a562bc75443701eec18a1f9726e"} Nov 03 21:01:23.038982 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:23.038950 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx" event={"ID":"36f2d843-7d6a-4606-b6b3-6904a2b6cc29","Type":"ContainerStarted","Data":"67144fb97d0431f7c086651fcdb49927a0bae010aebbd211c9419b7c124021fc"} Nov 03 21:01:23.041872 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:23.041124 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31b20f9f-bbs4g" event={"ID":"2806e408-4be8-4af6-90b3-4c122a06b402","Type":"ContainerDied","Data":"4dc644d7f5a54bb1048f24c2a278a103c8f111c086ac1083f7d3489981e97238"} Nov 03 21:01:23.041872 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:23.041169 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4dc644d7f5a54bb1048f24c2a278a103c8f111c086ac1083f7d3489981e97238" Nov 03 21:01:23.041872 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:23.041119 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31b20f9f-bbs4g" Nov 03 21:01:23.052653 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:23.052612 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v" event={"ID":"26613665-1067-4ba1-9b16-f84e8c583c2a","Type":"ContainerStarted","Data":"7a8de5f3446cbc61d84b9953de793ee6907da868f57f6cf4d763c54b45239cd1"} Nov 03 21:01:23.142423 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:23.142319 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44"] Nov 03 21:01:23.145699 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:01:23.145355 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dbbf182_5d1d_4a54_8f71_bb58afa654e1.slice/crio-2c5c72f54bd405be2f1f818e40d4610387d8b924e20b445be6a93c66ba266eb5 WatchSource:0}: Error finding container 2c5c72f54bd405be2f1f818e40d4610387d8b924e20b445be6a93c66ba266eb5: Status 404 returned error can't find the container with id 2c5c72f54bd405be2f1f818e40d4610387d8b924e20b445be6a93c66ba266eb5 Nov 03 21:01:23.228193 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:23.224418 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:01:23.228193 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:23.224590 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:01:23.228193 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:23.224687 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:01:23.248211 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:23.248147 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ffa5bc4-2lq5j"] Nov 03 21:01:23.294809 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:23.294657 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v" podStartSLOduration=1.294632981 podStartE2EDuration="1.294632981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:01:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:01:23.160212797 +0000 UTC m=+1318.385532605" watchObservedRunningTime="2026-11-03 21:01:23.294632981 +0000 UTC m=+1318.519952773" Nov 03 21:01:23.312817 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:23.312751 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx" podStartSLOduration=2.312735939 podStartE2EDuration="2.312735939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:01:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:01:23.309631001 +0000 UTC m=+1318.534950792" watchObservedRunningTime="2026-11-03 21:01:23.312735939 +0000 UTC m=+1318.538055729" Nov 03 21:01:23.491935 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:23.491732 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v" Nov 03 21:01:23.494641 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:23.494606 4305 patch_prober.go:29] interesting pod/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 03 21:01:23.494796 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:23.494694 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v" podUID="b24ff37e-1709-4f11-b0e8-9302b9f3d9ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 03 21:01:23.709514 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:23.709478 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx" Nov 03 21:01:23.711643 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:23.711613 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 03 21:01:23.711780 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:23.711696 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx" podUID="36f2d843-7d6a-4606-b6b3-6904a2b6cc29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 03 21:01:24.101461 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:24.098203 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b319d281-57hgb"] Nov 03 21:01:24.150473 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:24.147650 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b319d281-57hgb"] Nov 03 21:01:24.166473 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:24.165636 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ffa5bc4-2lq5j" event={"ID":"ce39a4a7-549b-4163-aab1-ac6581c9c451","Type":"ContainerStarted","Data":"429c663c84c04c97f21a4985e25896f2bd1437d74c43056a7f07400f67da94e7"} Nov 03 21:01:24.166473 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:24.165722 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ffa5bc4-2lq5j" event={"ID":"ce39a4a7-549b-4163-aab1-ac6581c9c451","Type":"ContainerStarted","Data":"50912f10999c532fe1e3fb192c8c9337e3601b75bfad289d2f342f4f3fd3cdc4"} Nov 03 21:01:24.189761 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:24.189719 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v" event={"ID":"26613665-1067-4ba1-9b16-f84e8c583c2a","Type":"ContainerStarted","Data":"bdd21d6639c9bd024bc58a627e4e6f119dda274b402f4315f27dc4baf0d8eb52"} Nov 03 21:01:24.242718 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:24.242682 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44" event={"ID":"4dbbf182-5d1d-4a54-8f71-bb58afa654e1","Type":"ContainerStarted","Data":"438f8ea599856ffa9ea59ba3f14e7c6de51041f43281cdf9c22cabe13785fd3f"} Nov 03 21:01:24.257026 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:24.256984 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44" event={"ID":"4dbbf182-5d1d-4a54-8f71-bb58afa654e1","Type":"ContainerStarted","Data":"2c5c72f54bd405be2f1f818e40d4610387d8b924e20b445be6a93c66ba266eb5"} Nov 03 21:01:24.308361 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:24.308289 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ffa5bc4-2lq5j" podStartSLOduration=2.3082619109999998 podStartE2EDuration="2.308261911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:01:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:01:24.297652453 +0000 UTC m=+1319.522972265" watchObservedRunningTime="2026-11-03 21:01:24.308261911 +0000 UTC m=+1319.533581703" Nov 03 21:01:24.436322 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:24.436250 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v" podStartSLOduration=2.43622549 podStartE2EDuration="2.43622549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:01:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:01:24.423979491 +0000 UTC m=+1319.649299282" watchObservedRunningTime="2026-11-03 21:01:24.43622549 +0000 UTC m=+1319.661545284" Nov 03 21:01:24.493896 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:24.493862 4305 patch_prober.go:29] interesting pod/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 03 21:01:24.494356 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:24.494323 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v" podUID="b24ff37e-1709-4f11-b0e8-9302b9f3d9ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 03 21:01:24.547037 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:24.544756 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v" Nov 03 21:01:24.547481 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:24.547450 4305 patch_prober.go:29] interesting pod/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 03 21:01:24.547754 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:24.547714 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v" podUID="26613665-1067-4ba1-9b16-f84e8c583c2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 03 21:01:24.715116 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:24.715014 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 03 21:01:24.715393 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:24.715352 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx" podUID="36f2d843-7d6a-4606-b6b3-6904a2b6cc29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 03 21:01:24.715860 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:24.715844 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44" Nov 03 21:01:24.718747 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:24.718721 4305 patch_prober.go:29] interesting pod/e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 03 21:01:24.718955 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:24.718934 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44" podUID="4dbbf182-5d1d-4a54-8f71-bb58afa654e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 03 21:01:25.026196 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:25.026092 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" podUID="793b8ca3-3a2e-49a1-9be4-4971ba59df09" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:25.050770 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:25.050735 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65e231f1-20dc-4c54-b7f1-4217ab1a0965" path="/var/lib/kubelet/pods/65e231f1-20dc-4c54-b7f1-4217ab1a0965/volumes" Nov 03 21:01:25.101463 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:25.091362 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" podUID="8793f095-97b1-4e0a-8382-b541f90a0de2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 03 21:01:25.101463 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:25.092090 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" podUID="63a704b4-51ce-46c3-8007-34b308aff332" containerName="main" probeResult="failure" output="Get \"https://10.134.0.101:8000/ready\": context deadline exceeded" Nov 03 21:01:25.121461 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:25.104191 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" podUID="6e7f30a9-0862-48a5-9ab7-e692e75bb294" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:25.182641 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:25.182582 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" podUID="d5d3dcaa-a51a-40c6-8049-871b300552e1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:25.183064 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:25.183022 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" podUID="8ee1f028-8503-4034-9954-eb06fcf402e7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": context deadline exceeded" Nov 03 21:01:25.183329 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:25.183305 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" podUID="7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 03 21:01:25.183615 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:25.183589 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" podUID="7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 03 21:01:25.183895 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:25.183865 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" podUID="be06bc96-808a-46e6-9872-01baf1a11a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.107:8000/ready\": context deadline exceeded" Nov 03 21:01:25.269592 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:25.255285 4305 scope.go:122] "RemoveContainer" containerID="78ce158ced8712488aa431a6f61b504347fc7656eba0c26026f89aa38ebcc2ca" Nov 03 21:01:25.521464 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:25.503200 4305 patch_prober.go:29] interesting pod/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 03 21:01:25.521464 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:25.503263 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v" podUID="b24ff37e-1709-4f11-b0e8-9302b9f3d9ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 03 21:01:25.521464 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:25.518410 4305 generic.go:356] "Generic (PLEG): container finished" podID="ce39a4a7-549b-4163-aab1-ac6581c9c451" containerID="429c663c84c04c97f21a4985e25896f2bd1437d74c43056a7f07400f67da94e7" exitCode=0 Nov 03 21:01:25.521464 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:25.520001 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ffa5bc4-2lq5j" event={"ID":"ce39a4a7-549b-4163-aab1-ac6581c9c451","Type":"ContainerDied","Data":"429c663c84c04c97f21a4985e25896f2bd1437d74c43056a7f07400f67da94e7"} Nov 03 21:01:25.563460 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:25.561372 4305 patch_prober.go:29] interesting pod/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 03 21:01:25.563460 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:25.561465 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v" podUID="26613665-1067-4ba1-9b16-f84e8c583c2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 03 21:01:25.626473 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:25.601663 4305 scope.go:122] "RemoveContainer" containerID="86b9f07108518c07b99ae7e0e4870772ab608287c789bec95617e7e23057206b" Nov 03 21:01:25.682461 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:25.682238 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44" podStartSLOduration=3.682217471 podStartE2EDuration="3.682217471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:01:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:01:24.607883795 +0000 UTC m=+1319.833203604" watchObservedRunningTime="2026-11-03 21:01:25.682217471 +0000 UTC m=+1320.907537262" Nov 03 21:01:25.724518 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:25.712287 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 03 21:01:25.724518 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:25.712356 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx" podUID="36f2d843-7d6a-4606-b6b3-6904a2b6cc29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 03 21:01:25.724518 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:25.712836 4305 patch_prober.go:29] interesting pod/e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 03 21:01:25.724518 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:25.712880 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44" podUID="4dbbf182-5d1d-4a54-8f71-bb58afa654e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 03 21:01:25.744307 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:25.744246 4305 scope.go:122] "RemoveContainer" containerID="bc4ea3368e5bf7d3eb45b0b4f1ba45fdb232ee51835d1deb8cbbf75cf5b3eb9c" Nov 03 21:01:26.035539 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:26.034980 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" podUID="80de7944-70f6-49e1-a0e8-4b82d5caad0d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:26.062621 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:26.062571 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" podUID="006c5eae-06af-447b-a1d8-cd719cce2a92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:26.063971 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:26.063919 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" podUID="793b8ca3-3a2e-49a1-9be4-4971ba59df09" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:26.064587 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:26.064558 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" Nov 03 21:01:26.092584 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:26.092537 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" podUID="63a704b4-51ce-46c3-8007-34b308aff332" containerName="main" probeResult="failure" output="Get \"https://10.134.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:26.096839 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:26.095793 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" podUID="8793f095-97b1-4e0a-8382-b541f90a0de2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 03 21:01:26.097455 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:26.097410 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" Nov 03 21:01:26.122464 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:26.097651 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" Nov 03 21:01:26.123839 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:26.122594 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" podUID="6e7f30a9-0862-48a5-9ab7-e692e75bb294" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:26.124532 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:26.124509 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" Nov 03 21:01:26.289525 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:26.211606 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" podUID="be06bc96-808a-46e6-9872-01baf1a11a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:26.344284 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:26.216017 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" podUID="8ee1f028-8503-4034-9954-eb06fcf402e7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": context deadline exceeded" Nov 03 21:01:26.344909 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:26.344886 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" Nov 03 21:01:26.345119 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:26.345108 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" Nov 03 21:01:26.345625 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:26.216129 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" podUID="d5d3dcaa-a51a-40c6-8049-871b300552e1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded" Nov 03 21:01:26.346415 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:26.346397 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" Nov 03 21:01:26.500460 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:26.492628 4305 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" podUID="8ee1f028-8503-4034-9954-eb06fcf402e7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:26.766458 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:26.738000 4305 patch_prober.go:29] interesting pod/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 03 21:01:26.770459 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:26.767314 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v" podUID="26613665-1067-4ba1-9b16-f84e8c583c2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 03 21:01:26.770459 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:26.764578 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" Nov 03 21:01:26.803698 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:26.803663 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 03 21:01:26.804001 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:26.803958 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx" podUID="36f2d843-7d6a-4606-b6b3-6904a2b6cc29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 03 21:01:26.807809 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:26.807780 4305 patch_prober.go:29] interesting pod/e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 03 21:01:26.808106 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:26.808064 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44" podUID="4dbbf182-5d1d-4a54-8f71-bb58afa654e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 03 21:01:26.831344 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:26.808544 4305 patch_prober.go:29] interesting pod/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 03 21:01:26.831344 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:26.808636 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v" podUID="b24ff37e-1709-4f11-b0e8-9302b9f3d9ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 03 21:01:27.164615 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:27.164558 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" podUID="80de7944-70f6-49e1-a0e8-4b82d5caad0d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Nov 03 21:01:27.166257 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:27.166218 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" podUID="793b8ca3-3a2e-49a1-9be4-4971ba59df09" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 03 21:01:27.170228 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:27.170186 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" podUID="8793f095-97b1-4e0a-8382-b541f90a0de2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:27.174770 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:27.174716 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" podUID="006c5eae-06af-447b-a1d8-cd719cce2a92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:27.197598 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:27.184061 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" podUID="6e7f30a9-0862-48a5-9ab7-e692e75bb294" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:27.197598 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:27.184169 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" podUID="7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:27.216468 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:27.215827 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" podUID="7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:27.345599 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:27.345549 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" podUID="8ee1f028-8503-4034-9954-eb06fcf402e7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:27.378464 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:27.353536 4305 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" podUID="8793f095-97b1-4e0a-8382-b541f90a0de2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": context deadline exceeded" Nov 03 21:01:27.378464 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:27.357921 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" podUID="d5d3dcaa-a51a-40c6-8049-871b300552e1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:27.378464 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:27.359065 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" podUID="be06bc96-808a-46e6-9872-01baf1a11a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.107:8000/ready\": context deadline exceeded" Nov 03 21:01:27.565462 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:27.561785 4305 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" podUID="80de7944-70f6-49e1-a0e8-4b82d5caad0d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:27.565462 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:27.562707 4305 patch_prober.go:29] interesting pod/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 03 21:01:27.565462 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:27.562752 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v" podUID="26613665-1067-4ba1-9b16-f84e8c583c2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 03 21:01:27.565462 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:27.562829 4305 patch_prober.go:29] interesting pod/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 03 21:01:27.565462 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:27.562850 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v" podUID="b24ff37e-1709-4f11-b0e8-9302b9f3d9ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 03 21:01:27.629576 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:27.613210 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9311f42a-6pdhs"] Nov 03 21:01:27.629889 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:27.629859 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9311f42a-6pdhs"] Nov 03 21:01:27.724458 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:27.712882 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" Nov 03 21:01:27.724458 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:27.712947 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" Nov 03 21:01:27.773930 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:27.771888 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" podUID="63a704b4-51ce-46c3-8007-34b308aff332" containerName="main" probeResult="failure" output="Get \"https://10.134.0.101:8000/ready\": context deadline exceeded" Nov 03 21:01:27.900817 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:27.844592 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 03 21:01:27.900817 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:27.844653 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx" podUID="36f2d843-7d6a-4606-b6b3-6904a2b6cc29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 03 21:01:27.900817 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:27.844732 4305 patch_prober.go:29] interesting pod/e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 03 21:01:27.900817 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:27.844752 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44" podUID="4dbbf182-5d1d-4a54-8f71-bb58afa654e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 03 21:01:27.919474 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:27.914868 4305 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" podUID="63a704b4-51ce-46c3-8007-34b308aff332" containerName="main" probeResult="failure" output="Get \"https://10.134.0.101:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:28.172461 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:28.166586 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" podUID="80de7944-70f6-49e1-a0e8-4b82d5caad0d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:28.175109 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:28.172898 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" podUID="793b8ca3-3a2e-49a1-9be4-4971ba59df09" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:28.348692 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:28.348643 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" podUID="6e7f30a9-0862-48a5-9ab7-e692e75bb294" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:28.360477 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:28.352618 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" podUID="8ee1f028-8503-4034-9954-eb06fcf402e7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": context deadline exceeded" Nov 03 21:01:28.369472 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:28.360806 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" podUID="be06bc96-808a-46e6-9872-01baf1a11a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:28.373455 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:28.360068 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" podUID="7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:28.373455 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:28.360925 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" podUID="7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 03 21:01:28.373455 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:28.370515 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" Nov 03 21:01:28.373455 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:28.370583 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" Nov 03 21:01:28.373455 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:28.360033 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" podUID="006c5eae-06af-447b-a1d8-cd719cce2a92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:28.373455 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:28.361599 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" podUID="d5d3dcaa-a51a-40c6-8049-871b300552e1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded" Nov 03 21:01:28.430520 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:28.361626 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" podUID="8793f095-97b1-4e0a-8382-b541f90a0de2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:28.630460 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:28.533129 4305 patch_prober.go:29] interesting pod/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 03 21:01:28.630460 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:28.533191 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v" podUID="b24ff37e-1709-4f11-b0e8-9302b9f3d9ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 03 21:01:28.630460 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:28.549033 4305 patch_prober.go:29] interesting pod/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 03 21:01:28.630460 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:28.549088 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v" podUID="26613665-1067-4ba1-9b16-f84e8c583c2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 03 21:01:28.782527 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:28.777776 4305 patch_prober.go:29] interesting pod/e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 03 21:01:28.782527 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:28.777832 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44" podUID="4dbbf182-5d1d-4a54-8f71-bb58afa654e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 03 21:01:28.782527 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:28.778313 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 03 21:01:28.782527 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:28.778351 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx" podUID="36f2d843-7d6a-4606-b6b3-6904a2b6cc29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 03 21:01:28.812710 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:28.811187 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" podUID="63a704b4-51ce-46c3-8007-34b308aff332" containerName="main" probeResult="failure" output="Get \"https://10.134.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:28.812710 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:28.811681 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" Nov 03 21:01:29.079545 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:29.074977 4305 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" podUID="793b8ca3-3a2e-49a1-9be4-4971ba59df09" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:29.182726 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:29.182679 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" podUID="793b8ca3-3a2e-49a1-9be4-4971ba59df09" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:29.183946 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:29.183903 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" podUID="80de7944-70f6-49e1-a0e8-4b82d5caad0d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:29.241459 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:29.231279 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17a26a30-1c68-44c7-ac11-2269b4d2b3d4" path="/var/lib/kubelet/pods/17a26a30-1c68-44c7-ac11-2269b4d2b3d4/volumes" Nov 03 21:01:29.315996 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:29.315855 4305 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" podUID="6e7f30a9-0862-48a5-9ab7-e692e75bb294" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/health\": context deadline exceeded" Nov 03 21:01:29.356549 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:29.355728 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" podUID="6e7f30a9-0862-48a5-9ab7-e692e75bb294" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:29.387489 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:29.374543 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" podUID="d5d3dcaa-a51a-40c6-8049-871b300552e1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:29.387489 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:29.374953 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" podUID="8ee1f028-8503-4034-9954-eb06fcf402e7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": context deadline exceeded" Nov 03 21:01:29.387489 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:29.375222 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" podUID="006c5eae-06af-447b-a1d8-cd719cce2a92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:29.387489 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:29.375497 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" podUID="7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:29.387489 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:29.375737 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" podUID="7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 03 21:01:29.424461 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:29.422260 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" podUID="be06bc96-808a-46e6-9872-01baf1a11a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.107:8000/ready\": context deadline exceeded" Nov 03 21:01:29.476460 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:29.443655 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" podUID="8793f095-97b1-4e0a-8382-b541f90a0de2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:29.541649 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:29.541592 4305 patch_prober.go:29] interesting pod/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 03 21:01:29.546464 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:29.541949 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v" podUID="b24ff37e-1709-4f11-b0e8-9302b9f3d9ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 03 21:01:29.570462 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:29.563500 4305 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" podUID="be06bc96-808a-46e6-9872-01baf1a11a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.107:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:29.578853 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:29.578808 4305 patch_prober.go:29] interesting pod/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 03 21:01:29.624719 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:29.624598 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v" podUID="26613665-1067-4ba1-9b16-f84e8c583c2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 03 21:01:29.804508 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:29.804469 4305 patch_prober.go:29] interesting pod/e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 03 21:01:29.804837 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:29.804792 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44" podUID="4dbbf182-5d1d-4a54-8f71-bb58afa654e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 03 21:01:29.858697 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:29.811605 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" podUID="63a704b4-51ce-46c3-8007-34b308aff332" containerName="main" probeResult="failure" output="Get \"https://10.134.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:29.858697 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:29.816605 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 03 21:01:29.858697 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:29.816654 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx" podUID="36f2d843-7d6a-4606-b6b3-6904a2b6cc29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 03 21:01:30.152473 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:30.149632 4305 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" podUID="7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:30.213494 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:30.204028 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" podUID="793b8ca3-3a2e-49a1-9be4-4971ba59df09" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:30.313718 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:30.242362 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" podUID="80de7944-70f6-49e1-a0e8-4b82d5caad0d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:30.356997 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:30.356551 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" podUID="6e7f30a9-0862-48a5-9ab7-e692e75bb294" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:30.384807 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:30.384707 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be167f9a-cldkf"] Nov 03 21:01:30.389226 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:30.389184 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" podUID="7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:30.410459 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:30.410299 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" podUID="d5d3dcaa-a51a-40c6-8049-871b300552e1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded" Nov 03 21:01:30.434601 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:30.410508 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" podUID="8ee1f028-8503-4034-9954-eb06fcf402e7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": context deadline exceeded" Nov 03 21:01:30.434601 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:30.410784 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" podUID="006c5eae-06af-447b-a1d8-cd719cce2a92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Nov 03 21:01:30.434601 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:30.411001 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" podUID="7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:30.434601 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:30.429620 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" podUID="be06bc96-808a-46e6-9872-01baf1a11a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:30.473291 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:30.450073 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" podUID="8793f095-97b1-4e0a-8382-b541f90a0de2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:30.511461 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:30.508707 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71cb34d9-lmbgq"] Nov 03 21:01:30.613456 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:30.611630 4305 patch_prober.go:29] interesting pod/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 03 21:01:30.613456 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:30.611707 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v" podUID="b24ff37e-1709-4f11-b0e8-9302b9f3d9ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 03 21:01:30.735182 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:30.735088 4305 patch_prober.go:29] interesting pod/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 03 21:01:30.743493 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:30.735399 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v" podUID="26613665-1067-4ba1-9b16-f84e8c583c2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 03 21:01:30.811133 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:30.741119 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" Nov 03 21:01:30.901854 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:30.901722 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" podUID="63a704b4-51ce-46c3-8007-34b308aff332" containerName="main" probeResult="failure" output="Get \"https://10.134.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:30.916805 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:30.916493 4305 patch_prober.go:29] interesting pod/e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 03 21:01:30.921143 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:30.921105 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44" podUID="4dbbf182-5d1d-4a54-8f71-bb58afa654e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 03 21:01:30.921371 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:30.918054 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" Nov 03 21:01:30.921676 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:30.921642 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 03 21:01:30.921830 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:30.921811 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx" podUID="36f2d843-7d6a-4606-b6b3-6904a2b6cc29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 03 21:01:30.929528 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:30.924143 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" Nov 03 21:01:30.950580 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:30.950552 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be167f9a-cldkf"] Nov 03 21:01:31.003554 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:30.961836 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" Nov 03 21:01:31.012510 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:31.012423 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" Nov 03 21:01:31.142461 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:31.140886 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ffa5bc4-2lq5j" Nov 03 21:01:31.316029 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:31.315920 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" podUID="793b8ca3-3a2e-49a1-9be4-4971ba59df09" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:31.437569 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:31.435662 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" podUID="d5d3dcaa-a51a-40c6-8049-871b300552e1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded" Nov 03 21:01:31.576518 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:31.548783 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" podUID="8793f095-97b1-4e0a-8382-b541f90a0de2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 03 21:01:31.614461 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:31.597577 4305 patch_prober.go:29] interesting pod/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 03 21:01:31.614461 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:31.597645 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v" podUID="b24ff37e-1709-4f11-b0e8-9302b9f3d9ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 03 21:01:31.625021 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:31.624985 4305 patch_prober.go:29] interesting pod/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Nov 03 21:01:31.637571 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:31.637530 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v" podUID="26613665-1067-4ba1-9b16-f84e8c583c2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Nov 03 21:01:31.671859 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:31.633116 4305 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" podUID="006c5eae-06af-447b-a1d8-cd719cce2a92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:31.759465 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:31.732669 4305 patch_prober.go:29] interesting pod/e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 03 21:01:31.759465 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:31.732731 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44" podUID="4dbbf182-5d1d-4a54-8f71-bb58afa654e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 03 21:01:31.890561 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:31.853729 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60a33d48-25bd-4b1c-91da-911e825718c6" path="/var/lib/kubelet/pods/60a33d48-25bd-4b1c-91da-911e825718c6/volumes" Nov 03 21:01:31.890561 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:31.866743 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce39a4a7-549b-4163-aab1-ac6581c9c451-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce39a4a7-549b-4163-aab1-ac6581c9c451-maas-api-xks-ca\") pod \"ce39a4a7-549b-4163-aab1-ac6581c9c451\" (UID: \"ce39a4a7-549b-4163-aab1-ac6581c9c451\") " Nov 03 21:01:31.890561 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:31.866951 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce39a4a7-549b-4163-aab1-ac6581c9c451-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce39a4a7-549b-4163-aab1-ac6581c9c451-maas-api-openshift-ca\") pod \"ce39a4a7-549b-4163-aab1-ac6581c9c451\" (UID: \"ce39a4a7-549b-4163-aab1-ac6581c9c451\") " Nov 03 21:01:31.890561 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:31.867108 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce39a4a7-549b-4163-aab1-ac6581c9c451-kube-api-access-cwtx9\" (UniqueName: \"kubernetes.io/projected/ce39a4a7-549b-4163-aab1-ac6581c9c451-kube-api-access-cwtx9\") pod \"ce39a4a7-549b-4163-aab1-ac6581c9c451\" (UID: \"ce39a4a7-549b-4163-aab1-ac6581c9c451\") " Nov 03 21:01:31.890561 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:31.868605 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71cb34d9-lmbgq"] Nov 03 21:01:31.890561 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:31.868678 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" Nov 03 21:01:31.890561 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:31.868716 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" Nov 03 21:01:31.890561 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:31.868749 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" Nov 03 21:01:31.890561 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:31.868781 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" Nov 03 21:01:31.904369 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:31.904334 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce39a4a7-549b-4163-aab1-ac6581c9c451-maas-api-openshift-ca" pod "ce39a4a7-549b-4163-aab1-ac6581c9c451" (UID: "ce39a4a7-549b-4163-aab1-ac6581c9c451"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:01:32.126956 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:32.126919 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce39a4a7-549b-4163-aab1-ac6581c9c451-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:01:32.153638 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:32.134587 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" Nov 03 21:01:32.226372 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:32.226343 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" Nov 03 21:01:32.341461 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:32.335882 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce39a4a7-549b-4163-aab1-ac6581c9c451-maas-api-xks-ca" pod "ce39a4a7-549b-4163-aab1-ac6581c9c451" (UID: "ce39a4a7-549b-4163-aab1-ac6581c9c451"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:01:32.341461 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:32.337727 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ffa5bc4-2lq5j" event={"ID":"ce39a4a7-549b-4163-aab1-ac6581c9c451","Type":"ContainerDied","Data":"50912f10999c532fe1e3fb192c8c9337e3601b75bfad289d2f342f4f3fd3cdc4"} Nov 03 21:01:32.341461 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:32.337789 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50912f10999c532fe1e3fb192c8c9337e3601b75bfad289d2f342f4f3fd3cdc4" Nov 03 21:01:32.341461 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:32.337871 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ffa5bc4-2lq5j" Nov 03 21:01:32.351413 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:32.351385 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce39a4a7-549b-4163-aab1-ac6581c9c451-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:01:32.475623 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:32.458373 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce39a4a7-549b-4163-aab1-ac6581c9c451-kube-api-access-cwtx9" pod "ce39a4a7-549b-4163-aab1-ac6581c9c451" (UID: "ce39a4a7-549b-4163-aab1-ac6581c9c451"). InnerVolumeSpecName "kube-api-access-cwtx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:01:32.475623 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:32.461260 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwtx9\" (UniqueName: \"kubernetes.io/projected/ce39a4a7-549b-4163-aab1-ac6581c9c451-kube-api-access-cwtx9\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:01:32.629462 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:32.565816 4305 patch_prober.go:29] interesting pod/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 03 21:01:32.629462 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:32.565874 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v" podUID="b24ff37e-1709-4f11-b0e8-9302b9f3d9ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 03 21:01:32.949488 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:32.924675 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:01:32.949488 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:32.924737 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx" podUID="36f2d843-7d6a-4606-b6b3-6904a2b6cc29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:33.290538 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:33.284520 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" podUID="793b8ca3-3a2e-49a1-9be4-4971ba59df09" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:33.303486 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:33.302932 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" podUID="8793f095-97b1-4e0a-8382-b541f90a0de2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:33.456461 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:33.430923 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" podUID="be06bc96-808a-46e6-9872-01baf1a11a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:33.458161 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:33.458118 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" podUID="d5d3dcaa-a51a-40c6-8049-871b300552e1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:33.464592 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:33.464541 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" podUID="7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 03 21:01:33.469269 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:33.465183 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" podUID="006c5eae-06af-447b-a1d8-cd719cce2a92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Nov 03 21:01:33.489076 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:33.479504 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" podUID="7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 03 21:01:33.573545 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:33.560309 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf7d3b6c-e2b5-4899-b04e-b78a14240c91" path="/var/lib/kubelet/pods/bf7d3b6c-e2b5-4899-b04e-b78a14240c91/volumes" Nov 03 21:01:33.674536 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:33.602551 4305 patch_prober.go:29] interesting pod/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 03 21:01:33.674859 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:33.649457 4305 patch_prober.go:29] interesting pod/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:01:33.675037 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:33.675010 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v" podUID="26613665-1067-4ba1-9b16-f84e8c583c2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:33.675359 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:33.674797 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v" podUID="b24ff37e-1709-4f11-b0e8-9302b9f3d9ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 03 21:01:33.898461 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:33.892525 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" podUID="63a704b4-51ce-46c3-8007-34b308aff332" containerName="main" probeResult="failure" output="Get \"https://10.134.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:33.986905 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:33.986847 4305 patch_prober.go:29] interesting pod/e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:01:33.989814 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:33.989765 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44" podUID="4dbbf182-5d1d-4a54-8f71-bb58afa654e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:34.005286 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:34.004508 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:01:34.005286 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:34.004576 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx" podUID="36f2d843-7d6a-4606-b6b3-6904a2b6cc29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:34.306522 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:34.278274 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" podUID="80de7944-70f6-49e1-a0e8-4b82d5caad0d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Nov 03 21:01:34.306522 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:34.286010 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" podUID="6e7f30a9-0862-48a5-9ab7-e692e75bb294" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 03 21:01:34.306522 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:34.300617 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" podUID="793b8ca3-3a2e-49a1-9be4-4971ba59df09" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:34.306522 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:34.302348 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" Nov 03 21:01:34.330463 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:34.306962 4305 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" podUID="7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:34.330463 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:34.318606 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" podUID="8793f095-97b1-4e0a-8382-b541f90a0de2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:34.330463 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:34.319084 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" Nov 03 21:01:34.371859 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:34.371788 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" podUID="8ee1f028-8503-4034-9954-eb06fcf402e7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": context deadline exceeded" Nov 03 21:01:34.469354 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:34.469301 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" podUID="7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:34.469784 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:34.469752 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" podUID="be06bc96-808a-46e6-9872-01baf1a11a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.107:8000/ready\": context deadline exceeded" Nov 03 21:01:34.470085 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:34.470057 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" podUID="d5d3dcaa-a51a-40c6-8049-871b300552e1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded" Nov 03 21:01:34.470453 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:34.470415 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" Nov 03 21:01:34.470538 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:34.470526 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" Nov 03 21:01:34.470605 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:34.470591 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" Nov 03 21:01:34.489496 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:34.484427 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" podUID="006c5eae-06af-447b-a1d8-cd719cce2a92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:34.489496 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:34.484898 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" Nov 03 21:01:34.540464 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:34.539273 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" podUID="7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 03 21:01:34.540464 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:34.539691 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" Nov 03 21:01:34.802415 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:34.788054 4305 patch_prober.go:29] interesting pod/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:01:34.802415 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:34.788174 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v" podUID="26613665-1067-4ba1-9b16-f84e8c583c2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:34.941759 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:34.930798 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" podUID="63a704b4-51ce-46c3-8007-34b308aff332" containerName="main" probeResult="failure" output="Get \"https://10.134.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:35.072041 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:35.071939 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:01:35.072538 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:35.072375 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx" podUID="36f2d843-7d6a-4606-b6b3-6904a2b6cc29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:35.090775 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:35.090739 4305 patch_prober.go:29] interesting pod/e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:01:35.091057 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:35.091028 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44" podUID="4dbbf182-5d1d-4a54-8f71-bb58afa654e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:35.278765 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:35.278714 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" podUID="80de7944-70f6-49e1-a0e8-4b82d5caad0d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Nov 03 21:01:35.318250 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:35.318036 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" podUID="6e7f30a9-0862-48a5-9ab7-e692e75bb294" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:35.320305 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:35.320254 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" podUID="8793f095-97b1-4e0a-8382-b541f90a0de2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:35.320875 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:35.320849 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" podUID="793b8ca3-3a2e-49a1-9be4-4971ba59df09" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 03 21:01:35.363583 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:35.350844 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" Nov 03 21:01:35.871469 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:35.860566 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44" Nov 03 21:01:35.871469 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:35.860657 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" Nov 03 21:01:35.871469 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:35.860696 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" Nov 03 21:01:35.871469 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:35.390885 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" podUID="8ee1f028-8503-4034-9954-eb06fcf402e7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:35.885483 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:35.469704 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" podUID="7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:35.916584 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:35.470298 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" podUID="d5d3dcaa-a51a-40c6-8049-871b300552e1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:36.016946 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:35.605795 4305 patch_prober.go:29] interesting pod/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:01:36.016946 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:36.016149 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v" podUID="b24ff37e-1709-4f11-b0e8-9302b9f3d9ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:36.069460 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:35.825264 4305 patch_prober.go:29] interesting pod/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:01:36.069460 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:36.062966 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v" podUID="26613665-1067-4ba1-9b16-f84e8c583c2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:36.069460 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:35.916487 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44" Nov 03 21:01:36.069460 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:36.063933 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" Nov 03 21:01:36.069460 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:35.550130 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" podUID="7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:36.069460 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:35.470348 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" podUID="be06bc96-808a-46e6-9872-01baf1a11a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.107:8000/ready\": context deadline exceeded" Nov 03 21:01:36.082687 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:35.959168 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" podUID="63a704b4-51ce-46c3-8007-34b308aff332" containerName="main" probeResult="failure" output="Get \"https://10.134.0.101:8000/ready\": context deadline exceeded" Nov 03 21:01:36.083697 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:35.525710 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" podUID="006c5eae-06af-447b-a1d8-cd719cce2a92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:36.109473 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:36.090768 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44" Nov 03 21:01:36.162736 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:36.162697 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:01:36.163081 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:36.163033 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx" podUID="36f2d843-7d6a-4606-b6b3-6904a2b6cc29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:36.290825 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:36.290779 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" podUID="80de7944-70f6-49e1-a0e8-4b82d5caad0d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:36.313646 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:36.313614 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a02f5a1-c72m7"] Nov 03 21:01:36.446534 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:36.442541 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" podUID="8793f095-97b1-4e0a-8382-b541f90a0de2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 03 21:01:36.446534 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:36.442968 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" podUID="6e7f30a9-0862-48a5-9ab7-e692e75bb294" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:36.573460 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:36.572671 4305 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" podUID="8ee1f028-8503-4034-9954-eb06fcf402e7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/health\": context deadline exceeded" Nov 03 21:01:36.646461 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:36.627130 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" podUID="793b8ca3-3a2e-49a1-9be4-4971ba59df09" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 03 21:01:36.719449 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:36.719337 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx" Nov 03 21:01:36.949461 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:36.885595 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" podUID="7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:37.003908 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:37.003808 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" podUID="d5d3dcaa-a51a-40c6-8049-871b300552e1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded" Nov 03 21:01:37.066217 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:37.066158 4305 patch_prober.go:29] interesting pod/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:01:37.066532 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:37.066490 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v" podUID="b24ff37e-1709-4f11-b0e8-9302b9f3d9ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:37.074620 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:37.074046 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" podUID="be06bc96-808a-46e6-9872-01baf1a11a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:37.098289 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:37.098244 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" podUID="63a704b4-51ce-46c3-8007-34b308aff332" containerName="main" probeResult="failure" output="Get \"https://10.134.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:37.114994 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:37.114574 4305 patch_prober.go:29] interesting pod/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:01:37.114994 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:37.114640 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v" podUID="26613665-1067-4ba1-9b16-f84e8c583c2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:37.122470 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:37.121721 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx" Nov 03 21:01:37.122470 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:37.121811 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" Nov 03 21:01:37.122470 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:37.121852 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" Nov 03 21:01:37.122470 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:37.121884 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" Nov 03 21:01:37.129791 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:37.123374 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" Nov 03 21:01:37.129791 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:37.123427 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" Nov 03 21:01:37.129791 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:37.123471 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a02f5a1-c72m7"] Nov 03 21:01:37.129791 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:37.123503 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2b3bf0e-6xhtq"] Nov 03 21:01:37.129791 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:37.123544 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" Nov 03 21:01:37.129791 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:37.123562 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2b3bf0e-6xhtq"] Nov 03 21:01:37.129791 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:37.123613 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx" Nov 03 21:01:37.166649 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:37.162963 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0dfa3141-kbdr2"] Nov 03 21:01:37.166649 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:37.164102 4305 cpu_manager.go:414] "containerMap: removing container" podUID="ce39a4a7-549b-4163-aab1-ac6581c9c451" containerName="revoke-keys" Nov 03 21:01:37.166649 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:37.164126 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce39a4a7-549b-4163-aab1-ac6581c9c451" containerName="revoke-keys" Nov 03 21:01:37.166649 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:37.164351 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce39a4a7-549b-4163-aab1-ac6581c9c451" containerName="revoke-keys" Nov 03 21:01:37.171891 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:37.171794 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dfa3141-kbdr2" Nov 03 21:01:37.184449 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:37.184309 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0dfa3141-kbdr2"] Nov 03 21:01:37.312876 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:37.309072 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:01:37.319653 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:37.319624 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77vtd\"" Nov 03 21:01:37.319962 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:37.319942 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:01:37.320165 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:37.320148 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:01:37.351810 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:37.350748 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" podUID="80de7944-70f6-49e1-a0e8-4b82d5caad0d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:37.364113 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:37.364069 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" Nov 03 21:01:37.435956 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:37.432346 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" Nov 03 21:01:37.435956 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:37.435191 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" Nov 03 21:01:37.455857 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:37.437854 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58c57b4a-f8ee-4c2a-b83a-84ec067f0be4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0dfa3141-kbdr2\" (UID: \"58c57b4a-f8ee-4c2a-b83a-84ec067f0be4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dfa3141-kbdr2" Nov 03 21:01:37.455857 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:37.455419 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" podUID="6e7f30a9-0862-48a5-9ab7-e692e75bb294" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:37.456697 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:37.456156 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" Nov 03 21:01:37.509128 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:37.497338 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v" Nov 03 21:01:37.513388 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:37.513343 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7zv7\" (UniqueName: \"kubernetes.io/projected/58c57b4a-f8ee-4c2a-b83a-84ec067f0be4-kube-api-access-l7zv7\") pod \"maas-api-revoke-sub-0dfa3141-kbdr2\" (UID: \"58c57b4a-f8ee-4c2a-b83a-84ec067f0be4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dfa3141-kbdr2" Nov 03 21:01:37.513885 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:37.513847 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58c57b4a-f8ee-4c2a-b83a-84ec067f0be4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0dfa3141-kbdr2\" (UID: \"58c57b4a-f8ee-4c2a-b83a-84ec067f0be4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dfa3141-kbdr2" Nov 03 21:01:37.546605 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:37.539598 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v" Nov 03 21:01:37.549855 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:37.549832 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" Nov 03 21:01:37.652458 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:37.647711 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58c57b4a-f8ee-4c2a-b83a-84ec067f0be4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0dfa3141-kbdr2\" (UID: \"58c57b4a-f8ee-4c2a-b83a-84ec067f0be4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dfa3141-kbdr2" Nov 03 21:01:37.652458 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:37.647908 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7zv7\" (UniqueName: \"kubernetes.io/projected/58c57b4a-f8ee-4c2a-b83a-84ec067f0be4-kube-api-access-l7zv7\") pod \"maas-api-revoke-sub-0dfa3141-kbdr2\" (UID: \"58c57b4a-f8ee-4c2a-b83a-84ec067f0be4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dfa3141-kbdr2" Nov 03 21:01:37.652458 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:37.648017 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58c57b4a-f8ee-4c2a-b83a-84ec067f0be4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0dfa3141-kbdr2\" (UID: \"58c57b4a-f8ee-4c2a-b83a-84ec067f0be4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dfa3141-kbdr2" Nov 03 21:01:37.652458 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:37.652272 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58c57b4a-f8ee-4c2a-b83a-84ec067f0be4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0dfa3141-kbdr2\" (UID: \"58c57b4a-f8ee-4c2a-b83a-84ec067f0be4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dfa3141-kbdr2" Nov 03 21:01:37.684460 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:37.683599 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58c57b4a-f8ee-4c2a-b83a-84ec067f0be4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0dfa3141-kbdr2\" (UID: \"58c57b4a-f8ee-4c2a-b83a-84ec067f0be4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dfa3141-kbdr2" Nov 03 21:01:37.725307 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:37.725272 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7zv7\" (UniqueName: \"kubernetes.io/projected/58c57b4a-f8ee-4c2a-b83a-84ec067f0be4-kube-api-access-l7zv7\") pod \"maas-api-revoke-sub-0dfa3141-kbdr2\" (UID: \"58c57b4a-f8ee-4c2a-b83a-84ec067f0be4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dfa3141-kbdr2" Nov 03 21:01:37.791061 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:37.791033 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v" Nov 03 21:01:37.798965 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:37.798920 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v" Nov 03 21:01:37.853135 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:37.853104 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dfa3141-kbdr2" Nov 03 21:01:38.107589 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:38.107549 4305 patch_prober.go:29] interesting pod/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:01:38.108099 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:38.107603 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v" podUID="b24ff37e-1709-4f11-b0e8-9302b9f3d9ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:38.308138 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:38.308049 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0dfa3141-kbdr2"] Nov 03 21:01:38.313493 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:01:38.313458 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58c57b4a_f8ee_4c2a_b83a_84ec067f0be4.slice/crio-f54246fcb8f88c07732231fa354cf335413863ef65f0a7eb7f99cd84d3eee5a9 WatchSource:0}: Error finding container f54246fcb8f88c07732231fa354cf335413863ef65f0a7eb7f99cd84d3eee5a9: Status 404 returned error can't find the container with id f54246fcb8f88c07732231fa354cf335413863ef65f0a7eb7f99cd84d3eee5a9 Nov 03 21:01:38.518485 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:38.516983 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v" Nov 03 21:01:38.518485 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:38.518257 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v" Nov 03 21:01:38.639479 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:38.639345 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dfa3141-kbdr2" event={"ID":"58c57b4a-f8ee-4c2a-b83a-84ec067f0be4","Type":"ContainerStarted","Data":"f54246fcb8f88c07732231fa354cf335413863ef65f0a7eb7f99cd84d3eee5a9"} Nov 03 21:01:38.902269 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:38.902230 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cdbecf8-2c06-48d9-8f80-9ca469289354" path="/var/lib/kubelet/pods/6cdbecf8-2c06-48d9-8f80-9ca469289354/volumes" Nov 03 21:01:38.903061 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:38.903036 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97f3c196-0376-41ae-8b28-930d728277dd" path="/var/lib/kubelet/pods/97f3c196-0376-41ae-8b28-930d728277dd/volumes" Nov 03 21:01:39.650871 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:39.650837 4305 generic.go:356] "Generic (PLEG): container finished" podID="58c57b4a-f8ee-4c2a-b83a-84ec067f0be4" containerID="2dad96b48251c930abeb21eaf534e4fe5f98fde708cd7c7c3d83192b2f4c1185" exitCode=0 Nov 03 21:01:39.651294 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:39.650919 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dfa3141-kbdr2" event={"ID":"58c57b4a-f8ee-4c2a-b83a-84ec067f0be4","Type":"ContainerDied","Data":"2dad96b48251c930abeb21eaf534e4fe5f98fde708cd7c7c3d83192b2f4c1185"} Nov 03 21:01:40.849099 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:40.849067 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dfa3141-kbdr2" Nov 03 21:01:40.925691 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:40.925593 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58c57b4a-f8ee-4c2a-b83a-84ec067f0be4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58c57b4a-f8ee-4c2a-b83a-84ec067f0be4-maas-api-openshift-ca\") pod \"58c57b4a-f8ee-4c2a-b83a-84ec067f0be4\" (UID: \"58c57b4a-f8ee-4c2a-b83a-84ec067f0be4\") " Nov 03 21:01:40.925877 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:40.925745 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58c57b4a-f8ee-4c2a-b83a-84ec067f0be4-kube-api-access-l7zv7\" (UniqueName: \"kubernetes.io/projected/58c57b4a-f8ee-4c2a-b83a-84ec067f0be4-kube-api-access-l7zv7\") pod \"58c57b4a-f8ee-4c2a-b83a-84ec067f0be4\" (UID: \"58c57b4a-f8ee-4c2a-b83a-84ec067f0be4\") " Nov 03 21:01:40.925877 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:40.925787 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58c57b4a-f8ee-4c2a-b83a-84ec067f0be4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58c57b4a-f8ee-4c2a-b83a-84ec067f0be4-maas-api-xks-ca\") pod \"58c57b4a-f8ee-4c2a-b83a-84ec067f0be4\" (UID: \"58c57b4a-f8ee-4c2a-b83a-84ec067f0be4\") " Nov 03 21:01:40.926017 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:40.925988 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58c57b4a-f8ee-4c2a-b83a-84ec067f0be4-maas-api-openshift-ca" pod "58c57b4a-f8ee-4c2a-b83a-84ec067f0be4" (UID: "58c57b4a-f8ee-4c2a-b83a-84ec067f0be4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:01:40.926497 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:40.926422 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58c57b4a-f8ee-4c2a-b83a-84ec067f0be4-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:01:40.928145 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:40.928111 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58c57b4a-f8ee-4c2a-b83a-84ec067f0be4-maas-api-xks-ca" pod "58c57b4a-f8ee-4c2a-b83a-84ec067f0be4" (UID: "58c57b4a-f8ee-4c2a-b83a-84ec067f0be4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:01:40.928250 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:40.928149 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58c57b4a-f8ee-4c2a-b83a-84ec067f0be4-kube-api-access-l7zv7" pod "58c57b4a-f8ee-4c2a-b83a-84ec067f0be4" (UID: "58c57b4a-f8ee-4c2a-b83a-84ec067f0be4"). InnerVolumeSpecName "kube-api-access-l7zv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:01:41.027948 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:41.027901 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7zv7\" (UniqueName: \"kubernetes.io/projected/58c57b4a-f8ee-4c2a-b83a-84ec067f0be4-kube-api-access-l7zv7\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:01:41.028130 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:41.027976 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58c57b4a-f8ee-4c2a-b83a-84ec067f0be4-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:01:41.673392 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:41.673366 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dfa3141-kbdr2" Nov 03 21:01:41.673663 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:41.673365 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dfa3141-kbdr2" event={"ID":"58c57b4a-f8ee-4c2a-b83a-84ec067f0be4","Type":"ContainerDied","Data":"f54246fcb8f88c07732231fa354cf335413863ef65f0a7eb7f99cd84d3eee5a9"} Nov 03 21:01:41.673663 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:41.673503 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f54246fcb8f88c07732231fa354cf335413863ef65f0a7eb7f99cd84d3eee5a9" Nov 03 21:01:41.909127 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:41.909091 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ded6da31-znppf"] Nov 03 21:01:41.909872 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:41.909850 4305 cpu_manager.go:414] "containerMap: removing container" podUID="58c57b4a-f8ee-4c2a-b83a-84ec067f0be4" containerName="revoke-keys" Nov 03 21:01:41.909872 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:41.909869 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58c57b4a-f8ee-4c2a-b83a-84ec067f0be4" containerName="revoke-keys" Nov 03 21:01:41.910024 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:41.910015 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="58c57b4a-f8ee-4c2a-b83a-84ec067f0be4" containerName="revoke-keys" Nov 03 21:01:41.913919 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:41.913897 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ded6da31-znppf" Nov 03 21:01:41.966412 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:41.966333 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:01:41.967245 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:41.966636 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:01:41.967245 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:41.966760 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:01:41.967416 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:41.966777 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77vtd\"" Nov 03 21:01:41.971000 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:41.970968 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ded6da31-znppf"] Nov 03 21:01:42.045554 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:42.045519 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkq7k\" (UniqueName: \"kubernetes.io/projected/0a822cf6-57c9-4941-9797-e4605586b4b0-kube-api-access-dkq7k\") pod \"maas-api-revoke-sub-ded6da31-znppf\" (UID: \"0a822cf6-57c9-4941-9797-e4605586b4b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ded6da31-znppf" Nov 03 21:01:42.045742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:42.045717 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a822cf6-57c9-4941-9797-e4605586b4b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ded6da31-znppf\" (UID: \"0a822cf6-57c9-4941-9797-e4605586b4b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ded6da31-znppf" Nov 03 21:01:42.045893 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:42.045876 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a822cf6-57c9-4941-9797-e4605586b4b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ded6da31-znppf\" (UID: \"0a822cf6-57c9-4941-9797-e4605586b4b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ded6da31-znppf" Nov 03 21:01:42.050421 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:42.050397 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc281286-gk97p"] Nov 03 21:01:42.055489 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:42.055461 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc281286-gk97p"] Nov 03 21:01:42.147542 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:42.147493 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkq7k\" (UniqueName: \"kubernetes.io/projected/0a822cf6-57c9-4941-9797-e4605586b4b0-kube-api-access-dkq7k\") pod \"maas-api-revoke-sub-ded6da31-znppf\" (UID: \"0a822cf6-57c9-4941-9797-e4605586b4b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ded6da31-znppf" Nov 03 21:01:42.147804 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:42.147591 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a822cf6-57c9-4941-9797-e4605586b4b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ded6da31-znppf\" (UID: \"0a822cf6-57c9-4941-9797-e4605586b4b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ded6da31-znppf" Nov 03 21:01:42.147804 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:42.147659 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a822cf6-57c9-4941-9797-e4605586b4b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ded6da31-znppf\" (UID: \"0a822cf6-57c9-4941-9797-e4605586b4b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ded6da31-znppf" Nov 03 21:01:42.148391 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:42.148366 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a822cf6-57c9-4941-9797-e4605586b4b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ded6da31-znppf\" (UID: \"0a822cf6-57c9-4941-9797-e4605586b4b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ded6da31-znppf" Nov 03 21:01:42.150229 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:42.150208 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a822cf6-57c9-4941-9797-e4605586b4b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ded6da31-znppf\" (UID: \"0a822cf6-57c9-4941-9797-e4605586b4b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ded6da31-znppf" Nov 03 21:01:42.209310 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:42.209276 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkq7k\" (UniqueName: \"kubernetes.io/projected/0a822cf6-57c9-4941-9797-e4605586b4b0-kube-api-access-dkq7k\") pod \"maas-api-revoke-sub-ded6da31-znppf\" (UID: \"0a822cf6-57c9-4941-9797-e4605586b4b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ded6da31-znppf" Nov 03 21:01:42.238552 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:42.231868 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ded6da31-znppf" Nov 03 21:01:42.636846 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:42.636800 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ded6da31-znppf"] Nov 03 21:01:42.637247 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:01:42.637218 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a822cf6_57c9_4941_9797_e4605586b4b0.slice/crio-35bb3e484a0230445ca77c24bd67ee40264643afbc76b3ebb805df75fc43b2c6 WatchSource:0}: Error finding container 35bb3e484a0230445ca77c24bd67ee40264643afbc76b3ebb805df75fc43b2c6: Status 404 returned error can't find the container with id 35bb3e484a0230445ca77c24bd67ee40264643afbc76b3ebb805df75fc43b2c6 Nov 03 21:01:42.684939 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:42.684907 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ded6da31-znppf" event={"ID":"0a822cf6-57c9-4941-9797-e4605586b4b0","Type":"ContainerStarted","Data":"35bb3e484a0230445ca77c24bd67ee40264643afbc76b3ebb805df75fc43b2c6"} Nov 03 21:01:42.897850 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:42.897816 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d986989-2d1b-428a-b124-ecbce2ec3943" path="/var/lib/kubelet/pods/0d986989-2d1b-428a-b124-ecbce2ec3943/volumes" Nov 03 21:01:43.455426 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:43.455390 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07445efa-87rzl"] Nov 03 21:01:43.459904 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:43.459882 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07445efa-87rzl" Nov 03 21:01:43.516545 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:43.516514 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07445efa-87rzl"] Nov 03 21:01:43.569973 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:43.569928 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50254126-8dc4-46f1-9a58-847f61765f36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07445efa-87rzl\" (UID: \"50254126-8dc4-46f1-9a58-847f61765f36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07445efa-87rzl" Nov 03 21:01:43.570207 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:43.570179 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnmfz\" (UniqueName: \"kubernetes.io/projected/50254126-8dc4-46f1-9a58-847f61765f36-kube-api-access-vnmfz\") pod \"maas-api-revoke-sub-07445efa-87rzl\" (UID: \"50254126-8dc4-46f1-9a58-847f61765f36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07445efa-87rzl" Nov 03 21:01:43.570320 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:43.570304 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50254126-8dc4-46f1-9a58-847f61765f36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07445efa-87rzl\" (UID: \"50254126-8dc4-46f1-9a58-847f61765f36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07445efa-87rzl" Nov 03 21:01:43.672201 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:43.672154 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50254126-8dc4-46f1-9a58-847f61765f36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07445efa-87rzl\" (UID: \"50254126-8dc4-46f1-9a58-847f61765f36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07445efa-87rzl" Nov 03 21:01:43.672392 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:43.672354 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnmfz\" (UniqueName: \"kubernetes.io/projected/50254126-8dc4-46f1-9a58-847f61765f36-kube-api-access-vnmfz\") pod \"maas-api-revoke-sub-07445efa-87rzl\" (UID: \"50254126-8dc4-46f1-9a58-847f61765f36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07445efa-87rzl" Nov 03 21:01:43.672510 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:43.672495 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50254126-8dc4-46f1-9a58-847f61765f36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07445efa-87rzl\" (UID: \"50254126-8dc4-46f1-9a58-847f61765f36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07445efa-87rzl" Nov 03 21:01:43.672961 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:43.672937 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50254126-8dc4-46f1-9a58-847f61765f36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07445efa-87rzl\" (UID: \"50254126-8dc4-46f1-9a58-847f61765f36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07445efa-87rzl" Nov 03 21:01:43.675036 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:43.675015 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50254126-8dc4-46f1-9a58-847f61765f36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07445efa-87rzl\" (UID: \"50254126-8dc4-46f1-9a58-847f61765f36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07445efa-87rzl" Nov 03 21:01:43.705134 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:43.705098 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ded6da31-znppf" event={"ID":"0a822cf6-57c9-4941-9797-e4605586b4b0","Type":"ContainerStarted","Data":"738c045b398f6baa821200ef859f3633882fc1656307ccd12549521f6f000bc3"} Nov 03 21:01:43.735519 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:43.735397 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnmfz\" (UniqueName: \"kubernetes.io/projected/50254126-8dc4-46f1-9a58-847f61765f36-kube-api-access-vnmfz\") pod \"maas-api-revoke-sub-07445efa-87rzl\" (UID: \"50254126-8dc4-46f1-9a58-847f61765f36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07445efa-87rzl" Nov 03 21:01:43.776749 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:43.776714 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07445efa-87rzl" Nov 03 21:01:43.824601 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:43.823168 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ded6da31-znppf" podStartSLOduration=2.823145167 podStartE2EDuration="2.823145167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:01:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:01:43.819739718 +0000 UTC m=+1339.045059508" watchObservedRunningTime="2026-11-03 21:01:43.823145167 +0000 UTC m=+1339.048464959" Nov 03 21:01:44.165886 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:44.165719 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07445efa-87rzl"] Nov 03 21:01:44.168738 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:01:44.168705 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50254126_8dc4_46f1_9a58_847f61765f36.slice/crio-b9c2f41366d9ecfb19af1e6bad8726919ba9667a58a6e4ee86e4c8bf4443b8c9 WatchSource:0}: Error finding container b9c2f41366d9ecfb19af1e6bad8726919ba9667a58a6e4ee86e4c8bf4443b8c9: Status 404 returned error can't find the container with id b9c2f41366d9ecfb19af1e6bad8726919ba9667a58a6e4ee86e4c8bf4443b8c9 Nov 03 21:01:44.714977 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:44.714944 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07445efa-87rzl" event={"ID":"50254126-8dc4-46f1-9a58-847f61765f36","Type":"ContainerStarted","Data":"16099414780cfc6f0e43de40711a4c98c7175da4ec768980d02847e39495412b"} Nov 03 21:01:44.715393 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:44.714998 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07445efa-87rzl" event={"ID":"50254126-8dc4-46f1-9a58-847f61765f36","Type":"ContainerStarted","Data":"b9c2f41366d9ecfb19af1e6bad8726919ba9667a58a6e4ee86e4c8bf4443b8c9"} Nov 03 21:01:44.716937 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:44.716881 4305 generic.go:356] "Generic (PLEG): container finished" podID="0a822cf6-57c9-4941-9797-e4605586b4b0" containerID="738c045b398f6baa821200ef859f3633882fc1656307ccd12549521f6f000bc3" exitCode=0 Nov 03 21:01:44.717167 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:44.717045 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ded6da31-znppf" event={"ID":"0a822cf6-57c9-4941-9797-e4605586b4b0","Type":"ContainerDied","Data":"738c045b398f6baa821200ef859f3633882fc1656307ccd12549521f6f000bc3"} Nov 03 21:01:44.871422 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:44.871314 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07445efa-87rzl" podStartSLOduration=1.871299752 podStartE2EDuration="1.871299752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:01:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:01:44.868338547 +0000 UTC m=+1340.093658358" watchObservedRunningTime="2026-11-03 21:01:44.871299752 +0000 UTC m=+1340.096619542" Nov 03 21:01:45.727012 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:45.726975 4305 generic.go:356] "Generic (PLEG): container finished" podID="50254126-8dc4-46f1-9a58-847f61765f36" containerID="16099414780cfc6f0e43de40711a4c98c7175da4ec768980d02847e39495412b" exitCode=0 Nov 03 21:01:45.727481 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:45.727064 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07445efa-87rzl" event={"ID":"50254126-8dc4-46f1-9a58-847f61765f36","Type":"ContainerDied","Data":"16099414780cfc6f0e43de40711a4c98c7175da4ec768980d02847e39495412b"} Nov 03 21:01:45.902160 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:45.902129 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ded6da31-znppf" Nov 03 21:01:46.009724 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:46.009626 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a822cf6-57c9-4941-9797-e4605586b4b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a822cf6-57c9-4941-9797-e4605586b4b0-maas-api-xks-ca\") pod \"0a822cf6-57c9-4941-9797-e4605586b4b0\" (UID: \"0a822cf6-57c9-4941-9797-e4605586b4b0\") " Nov 03 21:01:46.009724 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:46.009688 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a822cf6-57c9-4941-9797-e4605586b4b0-kube-api-access-dkq7k\" (UniqueName: \"kubernetes.io/projected/0a822cf6-57c9-4941-9797-e4605586b4b0-kube-api-access-dkq7k\") pod \"0a822cf6-57c9-4941-9797-e4605586b4b0\" (UID: \"0a822cf6-57c9-4941-9797-e4605586b4b0\") " Nov 03 21:01:46.009965 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:46.009746 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a822cf6-57c9-4941-9797-e4605586b4b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a822cf6-57c9-4941-9797-e4605586b4b0-maas-api-openshift-ca\") pod \"0a822cf6-57c9-4941-9797-e4605586b4b0\" (UID: \"0a822cf6-57c9-4941-9797-e4605586b4b0\") " Nov 03 21:01:46.010172 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:46.010146 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a822cf6-57c9-4941-9797-e4605586b4b0-maas-api-openshift-ca" pod "0a822cf6-57c9-4941-9797-e4605586b4b0" (UID: "0a822cf6-57c9-4941-9797-e4605586b4b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:01:46.010679 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:46.010658 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a822cf6-57c9-4941-9797-e4605586b4b0-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:01:46.012168 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:46.012130 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a822cf6-57c9-4941-9797-e4605586b4b0-maas-api-xks-ca" pod "0a822cf6-57c9-4941-9797-e4605586b4b0" (UID: "0a822cf6-57c9-4941-9797-e4605586b4b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:01:46.012168 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:46.012143 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a822cf6-57c9-4941-9797-e4605586b4b0-kube-api-access-dkq7k" pod "0a822cf6-57c9-4941-9797-e4605586b4b0" (UID: "0a822cf6-57c9-4941-9797-e4605586b4b0"). InnerVolumeSpecName "kube-api-access-dkq7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:01:46.112289 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:46.112251 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a822cf6-57c9-4941-9797-e4605586b4b0-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:01:46.112289 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:46.112285 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkq7k\" (UniqueName: \"kubernetes.io/projected/0a822cf6-57c9-4941-9797-e4605586b4b0-kube-api-access-dkq7k\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:01:46.325280 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:01:46.325189 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:01:46.736861 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:46.736826 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ded6da31-znppf" Nov 03 21:01:46.736861 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:46.736840 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ded6da31-znppf" event={"ID":"0a822cf6-57c9-4941-9797-e4605586b4b0","Type":"ContainerDied","Data":"35bb3e484a0230445ca77c24bd67ee40264643afbc76b3ebb805df75fc43b2c6"} Nov 03 21:01:46.737356 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:46.736930 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35bb3e484a0230445ca77c24bd67ee40264643afbc76b3ebb805df75fc43b2c6" Nov 03 21:01:46.935883 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:46.935857 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07445efa-87rzl" Nov 03 21:01:47.013565 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:47.009762 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-077e9e36-5jmj4"] Nov 03 21:01:47.013565 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:47.010844 4305 cpu_manager.go:414] "containerMap: removing container" podUID="50254126-8dc4-46f1-9a58-847f61765f36" containerName="revoke-keys" Nov 03 21:01:47.013565 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:47.010864 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50254126-8dc4-46f1-9a58-847f61765f36" containerName="revoke-keys" Nov 03 21:01:47.013565 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:47.010923 4305 cpu_manager.go:414] "containerMap: removing container" podUID="0a822cf6-57c9-4941-9797-e4605586b4b0" containerName="revoke-keys" Nov 03 21:01:47.013565 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:47.010930 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a822cf6-57c9-4941-9797-e4605586b4b0" containerName="revoke-keys" Nov 03 21:01:47.013565 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:47.011115 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="50254126-8dc4-46f1-9a58-847f61765f36" containerName="revoke-keys" Nov 03 21:01:47.013565 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:47.011137 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a822cf6-57c9-4941-9797-e4605586b4b0" containerName="revoke-keys" Nov 03 21:01:47.015803 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:47.015773 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-077e9e36-5jmj4" Nov 03 21:01:47.025232 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:47.025202 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50254126-8dc4-46f1-9a58-847f61765f36-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50254126-8dc4-46f1-9a58-847f61765f36-maas-api-xks-ca\") pod \"50254126-8dc4-46f1-9a58-847f61765f36\" (UID: \"50254126-8dc4-46f1-9a58-847f61765f36\") " Nov 03 21:01:47.025576 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:47.025541 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50254126-8dc4-46f1-9a58-847f61765f36-kube-api-access-vnmfz\" (UniqueName: \"kubernetes.io/projected/50254126-8dc4-46f1-9a58-847f61765f36-kube-api-access-vnmfz\") pod \"50254126-8dc4-46f1-9a58-847f61765f36\" (UID: \"50254126-8dc4-46f1-9a58-847f61765f36\") " Nov 03 21:01:47.028117 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:47.028080 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50254126-8dc4-46f1-9a58-847f61765f36-maas-api-xks-ca" pod "50254126-8dc4-46f1-9a58-847f61765f36" (UID: "50254126-8dc4-46f1-9a58-847f61765f36"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:01:47.028340 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:47.028319 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50254126-8dc4-46f1-9a58-847f61765f36-kube-api-access-vnmfz" pod "50254126-8dc4-46f1-9a58-847f61765f36" (UID: "50254126-8dc4-46f1-9a58-847f61765f36"). InnerVolumeSpecName "kube-api-access-vnmfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:01:47.098454 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:47.090553 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-077e9e36-5jmj4"] Nov 03 21:01:47.126938 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:47.126899 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50254126-8dc4-46f1-9a58-847f61765f36-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50254126-8dc4-46f1-9a58-847f61765f36-maas-api-openshift-ca\") pod \"50254126-8dc4-46f1-9a58-847f61765f36\" (UID: \"50254126-8dc4-46f1-9a58-847f61765f36\") " Nov 03 21:01:47.127284 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:47.127241 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50254126-8dc4-46f1-9a58-847f61765f36-maas-api-openshift-ca" pod "50254126-8dc4-46f1-9a58-847f61765f36" (UID: "50254126-8dc4-46f1-9a58-847f61765f36"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:01:47.127524 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:47.127502 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktcmv\" (UniqueName: \"kubernetes.io/projected/9e2d33ae-aad0-4600-a817-5c6f6d6771f7-kube-api-access-ktcmv\") pod \"maas-api-revoke-sub-077e9e36-5jmj4\" (UID: \"9e2d33ae-aad0-4600-a817-5c6f6d6771f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-077e9e36-5jmj4" Nov 03 21:01:47.127731 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:47.127709 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e2d33ae-aad0-4600-a817-5c6f6d6771f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-077e9e36-5jmj4\" (UID: \"9e2d33ae-aad0-4600-a817-5c6f6d6771f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-077e9e36-5jmj4" Nov 03 21:01:47.127935 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:47.127902 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e2d33ae-aad0-4600-a817-5c6f6d6771f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-077e9e36-5jmj4\" (UID: \"9e2d33ae-aad0-4600-a817-5c6f6d6771f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-077e9e36-5jmj4" Nov 03 21:01:47.128140 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:47.128122 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnmfz\" (UniqueName: \"kubernetes.io/projected/50254126-8dc4-46f1-9a58-847f61765f36-kube-api-access-vnmfz\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:01:47.128213 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:47.128146 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50254126-8dc4-46f1-9a58-847f61765f36-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:01:47.128213 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:47.128157 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50254126-8dc4-46f1-9a58-847f61765f36-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:01:47.229111 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:47.229071 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e2d33ae-aad0-4600-a817-5c6f6d6771f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-077e9e36-5jmj4\" (UID: \"9e2d33ae-aad0-4600-a817-5c6f6d6771f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-077e9e36-5jmj4" Nov 03 21:01:47.229297 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:47.229217 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e2d33ae-aad0-4600-a817-5c6f6d6771f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-077e9e36-5jmj4\" (UID: \"9e2d33ae-aad0-4600-a817-5c6f6d6771f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-077e9e36-5jmj4" Nov 03 21:01:47.229389 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:47.229373 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktcmv\" (UniqueName: \"kubernetes.io/projected/9e2d33ae-aad0-4600-a817-5c6f6d6771f7-kube-api-access-ktcmv\") pod \"maas-api-revoke-sub-077e9e36-5jmj4\" (UID: \"9e2d33ae-aad0-4600-a817-5c6f6d6771f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-077e9e36-5jmj4" Nov 03 21:01:47.229848 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:47.229823 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e2d33ae-aad0-4600-a817-5c6f6d6771f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-077e9e36-5jmj4\" (UID: \"9e2d33ae-aad0-4600-a817-5c6f6d6771f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-077e9e36-5jmj4" Nov 03 21:01:47.231886 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:47.231865 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e2d33ae-aad0-4600-a817-5c6f6d6771f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-077e9e36-5jmj4\" (UID: \"9e2d33ae-aad0-4600-a817-5c6f6d6771f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-077e9e36-5jmj4" Nov 03 21:01:47.290520 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:47.290410 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktcmv\" (UniqueName: \"kubernetes.io/projected/9e2d33ae-aad0-4600-a817-5c6f6d6771f7-kube-api-access-ktcmv\") pod \"maas-api-revoke-sub-077e9e36-5jmj4\" (UID: \"9e2d33ae-aad0-4600-a817-5c6f6d6771f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-077e9e36-5jmj4" Nov 03 21:01:47.333386 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:47.333350 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-077e9e36-5jmj4" Nov 03 21:01:47.722193 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:01:47.722153 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e2d33ae_aad0_4600_a817_5c6f6d6771f7.slice/crio-9533c137910bb0d6a6f7f07b1f6f5888b889ff698baffba700409b04b5105107 WatchSource:0}: Error finding container 9533c137910bb0d6a6f7f07b1f6f5888b889ff698baffba700409b04b5105107: Status 404 returned error can't find the container with id 9533c137910bb0d6a6f7f07b1f6f5888b889ff698baffba700409b04b5105107 Nov 03 21:01:47.747646 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:47.747604 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-077e9e36-5jmj4" event={"ID":"9e2d33ae-aad0-4600-a817-5c6f6d6771f7","Type":"ContainerStarted","Data":"9533c137910bb0d6a6f7f07b1f6f5888b889ff698baffba700409b04b5105107"} Nov 03 21:01:47.749292 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:47.749268 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07445efa-87rzl" Nov 03 21:01:47.749463 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:47.749272 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07445efa-87rzl" event={"ID":"50254126-8dc4-46f1-9a58-847f61765f36","Type":"ContainerDied","Data":"b9c2f41366d9ecfb19af1e6bad8726919ba9667a58a6e4ee86e4c8bf4443b8c9"} Nov 03 21:01:47.749463 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:47.749398 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9c2f41366d9ecfb19af1e6bad8726919ba9667a58a6e4ee86e4c8bf4443b8c9" Nov 03 21:01:47.769844 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:47.769813 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-077e9e36-5jmj4"] Nov 03 21:01:48.760051 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:48.760015 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-077e9e36-5jmj4" event={"ID":"9e2d33ae-aad0-4600-a817-5c6f6d6771f7","Type":"ContainerStarted","Data":"6c805e225e5f7e1244c5b662867c7baccd701cf49678c9cb1c5fab6f4f17c742"} Nov 03 21:01:48.879414 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:48.879355 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-077e9e36-5jmj4" podStartSLOduration=2.879340753 podStartE2EDuration="2.879340753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:01:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:01:48.874601545 +0000 UTC m=+1344.099921340" watchObservedRunningTime="2026-11-03 21:01:48.879340753 +0000 UTC m=+1344.104660575" Nov 03 21:01:49.769449 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:49.769407 4305 generic.go:356] "Generic (PLEG): container finished" podID="9e2d33ae-aad0-4600-a817-5c6f6d6771f7" containerID="6c805e225e5f7e1244c5b662867c7baccd701cf49678c9cb1c5fab6f4f17c742" exitCode=0 Nov 03 21:01:49.769896 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:49.769495 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-077e9e36-5jmj4" event={"ID":"9e2d33ae-aad0-4600-a817-5c6f6d6771f7","Type":"ContainerDied","Data":"6c805e225e5f7e1244c5b662867c7baccd701cf49678c9cb1c5fab6f4f17c742"} Nov 03 21:01:50.959952 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:50.959916 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-077e9e36-5jmj4" Nov 03 21:01:50.989231 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:50.989183 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e2d33ae-aad0-4600-a817-5c6f6d6771f7-kube-api-access-ktcmv\" (UniqueName: \"kubernetes.io/projected/9e2d33ae-aad0-4600-a817-5c6f6d6771f7-kube-api-access-ktcmv\") pod \"9e2d33ae-aad0-4600-a817-5c6f6d6771f7\" (UID: \"9e2d33ae-aad0-4600-a817-5c6f6d6771f7\") " Nov 03 21:01:50.989419 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:50.989397 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e2d33ae-aad0-4600-a817-5c6f6d6771f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e2d33ae-aad0-4600-a817-5c6f6d6771f7-maas-api-xks-ca\") pod \"9e2d33ae-aad0-4600-a817-5c6f6d6771f7\" (UID: \"9e2d33ae-aad0-4600-a817-5c6f6d6771f7\") " Nov 03 21:01:50.989589 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:50.989576 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e2d33ae-aad0-4600-a817-5c6f6d6771f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e2d33ae-aad0-4600-a817-5c6f6d6771f7-maas-api-openshift-ca\") pod \"9e2d33ae-aad0-4600-a817-5c6f6d6771f7\" (UID: \"9e2d33ae-aad0-4600-a817-5c6f6d6771f7\") " Nov 03 21:01:50.990054 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:50.989897 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e2d33ae-aad0-4600-a817-5c6f6d6771f7-maas-api-openshift-ca" pod "9e2d33ae-aad0-4600-a817-5c6f6d6771f7" (UID: "9e2d33ae-aad0-4600-a817-5c6f6d6771f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:01:50.990222 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:50.990203 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e2d33ae-aad0-4600-a817-5c6f6d6771f7-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:01:50.991659 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:50.991636 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e2d33ae-aad0-4600-a817-5c6f6d6771f7-kube-api-access-ktcmv" pod "9e2d33ae-aad0-4600-a817-5c6f6d6771f7" (UID: "9e2d33ae-aad0-4600-a817-5c6f6d6771f7"). InnerVolumeSpecName "kube-api-access-ktcmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:01:50.991767 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:50.991662 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e2d33ae-aad0-4600-a817-5c6f6d6771f7-maas-api-xks-ca" pod "9e2d33ae-aad0-4600-a817-5c6f6d6771f7" (UID: "9e2d33ae-aad0-4600-a817-5c6f6d6771f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:01:51.067079 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:51.066976 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-053094b3-fmfh2"] Nov 03 21:01:51.071956 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:51.071908 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38f52da4-r4b2r"] Nov 03 21:01:51.076697 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:51.076665 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-053094b3-fmfh2"] Nov 03 21:01:51.091505 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:51.091472 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktcmv\" (UniqueName: \"kubernetes.io/projected/9e2d33ae-aad0-4600-a817-5c6f6d6771f7-kube-api-access-ktcmv\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:01:51.091505 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:51.091504 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e2d33ae-aad0-4600-a817-5c6f6d6771f7-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:01:51.119638 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:51.119590 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38f52da4-r4b2r"] Nov 03 21:01:51.788116 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:51.788087 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-077e9e36-5jmj4" Nov 03 21:01:51.788307 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:51.788122 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-077e9e36-5jmj4" event={"ID":"9e2d33ae-aad0-4600-a817-5c6f6d6771f7","Type":"ContainerDied","Data":"9533c137910bb0d6a6f7f07b1f6f5888b889ff698baffba700409b04b5105107"} Nov 03 21:01:51.788307 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:51.788168 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9533c137910bb0d6a6f7f07b1f6f5888b889ff698baffba700409b04b5105107" Nov 03 21:01:52.209068 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:52.209032 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-59cjd"] Nov 03 21:01:52.209592 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:52.209298 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-59cjd" podUID="a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a" containerName="istio-proxy" containerID="cri-o://578e5a417d1a34d198a42091b4a70573ccb2ce9f030428a3cc36eb5db596531c" gracePeriod=30 Nov 03 21:01:52.258545 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:52.258502 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-588hz"] Nov 03 21:01:52.258759 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:52.258737 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-588hz" podUID="327b41fd-c8a7-4926-bc59-8869ae87aa8b" containerName="istio-proxy" containerID="cri-o://f423dd3aac35b5995843da8d11913875dfa9f8109624326e4f6739579de731d4" gracePeriod=30 Nov 03 21:01:52.898704 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:52.898663 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94566a28-565f-4d08-8178-ba8d3da0b963" path="/var/lib/kubelet/pods/94566a28-565f-4d08-8178-ba8d3da0b963/volumes" Nov 03 21:01:52.899404 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:52.899385 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7e2ad8b-2ee3-4fb4-8850-c6bf7d0f85e8" path="/var/lib/kubelet/pods/d7e2ad8b-2ee3-4fb4-8850-c6bf7d0f85e8/volumes" Nov 03 21:01:53.442515 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:53.442479 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14b52bda-77lqz"] Nov 03 21:01:53.443473 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:53.443413 4305 cpu_manager.go:414] "containerMap: removing container" podUID="9e2d33ae-aad0-4600-a817-5c6f6d6771f7" containerName="revoke-keys" Nov 03 21:01:53.443473 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:53.443449 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e2d33ae-aad0-4600-a817-5c6f6d6771f7" containerName="revoke-keys" Nov 03 21:01:53.443634 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:53.443608 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e2d33ae-aad0-4600-a817-5c6f6d6771f7" containerName="revoke-keys" Nov 03 21:01:53.452100 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:53.452075 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b52bda-77lqz" Nov 03 21:01:53.504849 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:53.504808 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:01:53.509447 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:53.509404 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14b52bda-77lqz"] Nov 03 21:01:53.534301 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:53.534255 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbtt5\" (UniqueName: \"kubernetes.io/projected/2fc27cc8-cbb7-4c99-95f7-1678fa27b72d-kube-api-access-lbtt5\") pod \"maas-api-revoke-sub-14b52bda-77lqz\" (UID: \"2fc27cc8-cbb7-4c99-95f7-1678fa27b72d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b52bda-77lqz" Nov 03 21:01:53.534536 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:53.534399 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fc27cc8-cbb7-4c99-95f7-1678fa27b72d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14b52bda-77lqz\" (UID: \"2fc27cc8-cbb7-4c99-95f7-1678fa27b72d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b52bda-77lqz" Nov 03 21:01:53.534616 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:53.534551 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fc27cc8-cbb7-4c99-95f7-1678fa27b72d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14b52bda-77lqz\" (UID: \"2fc27cc8-cbb7-4c99-95f7-1678fa27b72d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b52bda-77lqz" Nov 03 21:01:53.552961 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:53.552924 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:01:53.553179 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:53.552946 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77vtd\"" Nov 03 21:01:53.553179 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:53.552950 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:01:53.636402 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:53.636367 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fc27cc8-cbb7-4c99-95f7-1678fa27b72d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14b52bda-77lqz\" (UID: \"2fc27cc8-cbb7-4c99-95f7-1678fa27b72d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b52bda-77lqz" Nov 03 21:01:53.636646 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:53.636630 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbtt5\" (UniqueName: \"kubernetes.io/projected/2fc27cc8-cbb7-4c99-95f7-1678fa27b72d-kube-api-access-lbtt5\") pod \"maas-api-revoke-sub-14b52bda-77lqz\" (UID: \"2fc27cc8-cbb7-4c99-95f7-1678fa27b72d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b52bda-77lqz" Nov 03 21:01:53.636733 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:53.636687 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fc27cc8-cbb7-4c99-95f7-1678fa27b72d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14b52bda-77lqz\" (UID: \"2fc27cc8-cbb7-4c99-95f7-1678fa27b72d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b52bda-77lqz" Nov 03 21:01:53.637188 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:53.637163 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fc27cc8-cbb7-4c99-95f7-1678fa27b72d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14b52bda-77lqz\" (UID: \"2fc27cc8-cbb7-4c99-95f7-1678fa27b72d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b52bda-77lqz" Nov 03 21:01:53.639267 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:53.639240 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fc27cc8-cbb7-4c99-95f7-1678fa27b72d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14b52bda-77lqz\" (UID: \"2fc27cc8-cbb7-4c99-95f7-1678fa27b72d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b52bda-77lqz" Nov 03 21:01:53.697579 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:53.697502 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbtt5\" (UniqueName: \"kubernetes.io/projected/2fc27cc8-cbb7-4c99-95f7-1678fa27b72d-kube-api-access-lbtt5\") pod \"maas-api-revoke-sub-14b52bda-77lqz\" (UID: \"2fc27cc8-cbb7-4c99-95f7-1678fa27b72d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b52bda-77lqz" Nov 03 21:01:53.768092 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:53.768052 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b52bda-77lqz" Nov 03 21:01:54.159790 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:01:54.159757 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fc27cc8_cbb7_4c99_95f7_1678fa27b72d.slice/crio-623e82d7ec02a39df975e2c7cbc7bfa92e5739cdce86e3b1774a4a4295512c7c WatchSource:0}: Error finding container 623e82d7ec02a39df975e2c7cbc7bfa92e5739cdce86e3b1774a4a4295512c7c: Status 404 returned error can't find the container with id 623e82d7ec02a39df975e2c7cbc7bfa92e5739cdce86e3b1774a4a4295512c7c Nov 03 21:01:54.160610 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:54.160578 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14b52bda-77lqz"] Nov 03 21:01:54.830189 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:54.830089 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b52bda-77lqz" event={"ID":"2fc27cc8-cbb7-4c99-95f7-1678fa27b72d","Type":"ContainerStarted","Data":"06b88d529ebf9d878bb385fec6c6c36c3160cdc9588cdbfa25afd7e77a98a776"} Nov 03 21:01:54.830189 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:54.830176 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b52bda-77lqz" event={"ID":"2fc27cc8-cbb7-4c99-95f7-1678fa27b72d","Type":"ContainerStarted","Data":"623e82d7ec02a39df975e2c7cbc7bfa92e5739cdce86e3b1774a4a4295512c7c"} Nov 03 21:01:54.947074 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:54.947007 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b52bda-77lqz" podStartSLOduration=1.9469925460000002 podStartE2EDuration="1.946992546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:01:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:01:54.944812938 +0000 UTC m=+1350.170132749" watchObservedRunningTime="2026-11-03 21:01:54.946992546 +0000 UTC m=+1350.172312337" Nov 03 21:01:55.839812 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:55.839775 4305 generic.go:356] "Generic (PLEG): container finished" podID="2fc27cc8-cbb7-4c99-95f7-1678fa27b72d" containerID="06b88d529ebf9d878bb385fec6c6c36c3160cdc9588cdbfa25afd7e77a98a776" exitCode=0 Nov 03 21:01:55.840227 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:55.839859 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b52bda-77lqz" event={"ID":"2fc27cc8-cbb7-4c99-95f7-1678fa27b72d","Type":"ContainerDied","Data":"06b88d529ebf9d878bb385fec6c6c36c3160cdc9588cdbfa25afd7e77a98a776"} Nov 03 21:01:56.049162 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:56.049121 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6532ca59-p7h2d"] Nov 03 21:01:56.057033 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:56.057000 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6532ca59-p7h2d"] Nov 03 21:01:56.422878 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:56.422838 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fad5eaf-7tg2k"] Nov 03 21:01:56.427769 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:56.427742 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fad5eaf-7tg2k" Nov 03 21:01:56.482991 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:56.482871 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4c6f39d-5dcd-49a9-967b-4303f59d75c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9fad5eaf-7tg2k\" (UID: \"d4c6f39d-5dcd-49a9-967b-4303f59d75c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fad5eaf-7tg2k" Nov 03 21:01:56.483184 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:56.483167 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4c6f39d-5dcd-49a9-967b-4303f59d75c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9fad5eaf-7tg2k\" (UID: \"d4c6f39d-5dcd-49a9-967b-4303f59d75c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fad5eaf-7tg2k" Nov 03 21:01:56.483294 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:56.483276 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnr5w\" (UniqueName: \"kubernetes.io/projected/d4c6f39d-5dcd-49a9-967b-4303f59d75c0-kube-api-access-fnr5w\") pod \"maas-api-revoke-sub-9fad5eaf-7tg2k\" (UID: \"d4c6f39d-5dcd-49a9-967b-4303f59d75c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fad5eaf-7tg2k" Nov 03 21:01:56.484798 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:56.484771 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fad5eaf-7tg2k"] Nov 03 21:01:56.584956 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:56.584918 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4c6f39d-5dcd-49a9-967b-4303f59d75c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9fad5eaf-7tg2k\" (UID: \"d4c6f39d-5dcd-49a9-967b-4303f59d75c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fad5eaf-7tg2k" Nov 03 21:01:56.585198 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:56.585178 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4c6f39d-5dcd-49a9-967b-4303f59d75c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9fad5eaf-7tg2k\" (UID: \"d4c6f39d-5dcd-49a9-967b-4303f59d75c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fad5eaf-7tg2k" Nov 03 21:01:56.585312 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:56.585295 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnr5w\" (UniqueName: \"kubernetes.io/projected/d4c6f39d-5dcd-49a9-967b-4303f59d75c0-kube-api-access-fnr5w\") pod \"maas-api-revoke-sub-9fad5eaf-7tg2k\" (UID: \"d4c6f39d-5dcd-49a9-967b-4303f59d75c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fad5eaf-7tg2k" Nov 03 21:01:56.585973 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:56.585944 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4c6f39d-5dcd-49a9-967b-4303f59d75c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9fad5eaf-7tg2k\" (UID: \"d4c6f39d-5dcd-49a9-967b-4303f59d75c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fad5eaf-7tg2k" Nov 03 21:01:56.587698 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:56.587669 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4c6f39d-5dcd-49a9-967b-4303f59d75c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9fad5eaf-7tg2k\" (UID: \"d4c6f39d-5dcd-49a9-967b-4303f59d75c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fad5eaf-7tg2k" Nov 03 21:01:56.654401 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:56.654366 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnr5w\" (UniqueName: \"kubernetes.io/projected/d4c6f39d-5dcd-49a9-967b-4303f59d75c0-kube-api-access-fnr5w\") pod \"maas-api-revoke-sub-9fad5eaf-7tg2k\" (UID: \"d4c6f39d-5dcd-49a9-967b-4303f59d75c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fad5eaf-7tg2k" Nov 03 21:01:56.744856 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:56.744752 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fad5eaf-7tg2k" Nov 03 21:01:56.924517 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:56.923611 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bef6538-68a6-4d99-baa9-a7bf16dcb149" path="/var/lib/kubelet/pods/1bef6538-68a6-4d99-baa9-a7bf16dcb149/volumes" Nov 03 21:01:57.101538 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.101505 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b52bda-77lqz" Nov 03 21:01:57.142612 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:01:57.142576 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4c6f39d_5dcd_49a9_967b_4303f59d75c0.slice/crio-76d9227f84a915154775a2b33f5cf47f33b9d12cdbde7c251d2778e9e54ca9dd WatchSource:0}: Error finding container 76d9227f84a915154775a2b33f5cf47f33b9d12cdbde7c251d2778e9e54ca9dd: Status 404 returned error can't find the container with id 76d9227f84a915154775a2b33f5cf47f33b9d12cdbde7c251d2778e9e54ca9dd Nov 03 21:01:57.142612 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.142603 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fad5eaf-7tg2k"] Nov 03 21:01:57.208342 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.208314 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fc27cc8-cbb7-4c99-95f7-1678fa27b72d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fc27cc8-cbb7-4c99-95f7-1678fa27b72d-maas-api-openshift-ca\") pod \"2fc27cc8-cbb7-4c99-95f7-1678fa27b72d\" (UID: \"2fc27cc8-cbb7-4c99-95f7-1678fa27b72d\") " Nov 03 21:01:57.208540 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.208498 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fc27cc8-cbb7-4c99-95f7-1678fa27b72d-kube-api-access-lbtt5\" (UniqueName: \"kubernetes.io/projected/2fc27cc8-cbb7-4c99-95f7-1678fa27b72d-kube-api-access-lbtt5\") pod \"2fc27cc8-cbb7-4c99-95f7-1678fa27b72d\" (UID: \"2fc27cc8-cbb7-4c99-95f7-1678fa27b72d\") " Nov 03 21:01:57.208602 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.208580 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fc27cc8-cbb7-4c99-95f7-1678fa27b72d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fc27cc8-cbb7-4c99-95f7-1678fa27b72d-maas-api-xks-ca\") pod \"2fc27cc8-cbb7-4c99-95f7-1678fa27b72d\" (UID: \"2fc27cc8-cbb7-4c99-95f7-1678fa27b72d\") " Nov 03 21:01:57.208787 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.208762 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fc27cc8-cbb7-4c99-95f7-1678fa27b72d-maas-api-openshift-ca" pod "2fc27cc8-cbb7-4c99-95f7-1678fa27b72d" (UID: "2fc27cc8-cbb7-4c99-95f7-1678fa27b72d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:01:57.209204 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.209185 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fc27cc8-cbb7-4c99-95f7-1678fa27b72d-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:01:57.210906 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.210881 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fc27cc8-cbb7-4c99-95f7-1678fa27b72d-maas-api-xks-ca" pod "2fc27cc8-cbb7-4c99-95f7-1678fa27b72d" (UID: "2fc27cc8-cbb7-4c99-95f7-1678fa27b72d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:01:57.210983 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.210964 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fc27cc8-cbb7-4c99-95f7-1678fa27b72d-kube-api-access-lbtt5" pod "2fc27cc8-cbb7-4c99-95f7-1678fa27b72d" (UID: "2fc27cc8-cbb7-4c99-95f7-1678fa27b72d"). InnerVolumeSpecName "kube-api-access-lbtt5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:01:57.311011 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.310966 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbtt5\" (UniqueName: \"kubernetes.io/projected/2fc27cc8-cbb7-4c99-95f7-1678fa27b72d-kube-api-access-lbtt5\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:01:57.311011 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.311019 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fc27cc8-cbb7-4c99-95f7-1678fa27b72d-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:01:57.519703 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.519673 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-59cjd" Nov 03 21:01:57.614970 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.614939 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a-istio-data\") pod \"a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a\" (UID: \"a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a\") " Nov 03 21:01:57.615154 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.614981 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a-istio-token\" (UniqueName: \"kubernetes.io/projected/a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a-istio-token\") pod \"a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a\" (UID: \"a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a\") " Nov 03 21:01:57.615154 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.615010 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a-istio-envoy\") pod \"a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a\" (UID: \"a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a\") " Nov 03 21:01:57.615154 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.615049 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a-kube-api-access-bnxwt\" (UniqueName: \"kubernetes.io/projected/a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a-kube-api-access-bnxwt\") pod \"a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a\" (UID: \"a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a\") " Nov 03 21:01:57.615315 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.615175 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a-istio-podinfo\") pod \"a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a\" (UID: \"a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a\") " Nov 03 21:01:57.615315 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.615258 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a-workload-certs\") pod \"a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a\" (UID: \"a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a\") " Nov 03 21:01:57.615315 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.615267 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a-istio-data" pod "a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a" (UID: "a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:01:57.615494 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.615332 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a-istiod-ca-cert\") pod \"a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a\" (UID: \"a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a\") " Nov 03 21:01:57.615611 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.615573 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a-workload-socket\") pod \"a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a\" (UID: \"a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a\") " Nov 03 21:01:57.615611 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.615558 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a-workload-certs" pod "a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a" (UID: "a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:01:57.615716 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.615610 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a-credential-socket\") pod \"a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a\" (UID: \"a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a\") " Nov 03 21:01:57.615850 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.615706 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a-istiod-ca-cert" pod "a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a" (UID: "a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:01:57.615850 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.615787 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a-workload-socket" pod "a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a" (UID: "a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:01:57.616246 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.616225 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a-credential-socket" pod "a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a" (UID: "a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:01:57.616750 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.616725 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a-workload-certs\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:01:57.616856 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.616757 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a-istiod-ca-cert\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:01:57.616856 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.616773 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a-workload-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:01:57.616856 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.616788 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a-credential-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:01:57.616856 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.616802 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a-istio-data\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:01:57.658142 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.657133 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a-istio-podinfo" pod "a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a" (UID: "a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:01:57.658142 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.657166 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a-istio-envoy" pod "a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a" (UID: "a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:01:57.658142 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.657823 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a-istio-token" pod "a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a" (UID: "a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:01:57.658684 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.658654 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a-kube-api-access-bnxwt" pod "a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a" (UID: "a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a"). InnerVolumeSpecName "kube-api-access-bnxwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:01:57.694147 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.694116 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-588hz" Nov 03 21:01:57.718791 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.718756 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a-istio-token\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:01:57.718791 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.718791 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a-istio-envoy\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:01:57.719178 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.718831 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnxwt\" (UniqueName: \"kubernetes.io/projected/a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a-kube-api-access-bnxwt\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:01:57.719178 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.718848 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a-istio-podinfo\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:01:57.821913 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.821360 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/327b41fd-c8a7-4926-bc59-8869ae87aa8b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/327b41fd-c8a7-4926-bc59-8869ae87aa8b-credential-socket\") pod \"327b41fd-c8a7-4926-bc59-8869ae87aa8b\" (UID: \"327b41fd-c8a7-4926-bc59-8869ae87aa8b\") " Nov 03 21:01:57.822202 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.821479 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/327b41fd-c8a7-4926-bc59-8869ae87aa8b-credential-socket" pod "327b41fd-c8a7-4926-bc59-8869ae87aa8b" (UID: "327b41fd-c8a7-4926-bc59-8869ae87aa8b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:01:57.822324 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.822305 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/327b41fd-c8a7-4926-bc59-8869ae87aa8b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/327b41fd-c8a7-4926-bc59-8869ae87aa8b-workload-certs\") pod \"327b41fd-c8a7-4926-bc59-8869ae87aa8b\" (UID: \"327b41fd-c8a7-4926-bc59-8869ae87aa8b\") " Nov 03 21:01:57.822549 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.822534 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/327b41fd-c8a7-4926-bc59-8869ae87aa8b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/327b41fd-c8a7-4926-bc59-8869ae87aa8b-istio-data\") pod \"327b41fd-c8a7-4926-bc59-8869ae87aa8b\" (UID: \"327b41fd-c8a7-4926-bc59-8869ae87aa8b\") " Nov 03 21:01:57.822738 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.822597 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/327b41fd-c8a7-4926-bc59-8869ae87aa8b-workload-certs" pod "327b41fd-c8a7-4926-bc59-8869ae87aa8b" (UID: "327b41fd-c8a7-4926-bc59-8869ae87aa8b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:01:57.822878 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.822744 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/327b41fd-c8a7-4926-bc59-8869ae87aa8b-istio-data" pod "327b41fd-c8a7-4926-bc59-8869ae87aa8b" (UID: "327b41fd-c8a7-4926-bc59-8869ae87aa8b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:01:57.822878 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.822795 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/327b41fd-c8a7-4926-bc59-8869ae87aa8b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/327b41fd-c8a7-4926-bc59-8869ae87aa8b-istio-envoy\") pod \"327b41fd-c8a7-4926-bc59-8869ae87aa8b\" (UID: \"327b41fd-c8a7-4926-bc59-8869ae87aa8b\") " Nov 03 21:01:57.822878 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.822846 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/327b41fd-c8a7-4926-bc59-8869ae87aa8b-istio-token\" (UniqueName: \"kubernetes.io/projected/327b41fd-c8a7-4926-bc59-8869ae87aa8b-istio-token\") pod \"327b41fd-c8a7-4926-bc59-8869ae87aa8b\" (UID: \"327b41fd-c8a7-4926-bc59-8869ae87aa8b\") " Nov 03 21:01:57.823018 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.822899 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/327b41fd-c8a7-4926-bc59-8869ae87aa8b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/327b41fd-c8a7-4926-bc59-8869ae87aa8b-workload-socket\") pod \"327b41fd-c8a7-4926-bc59-8869ae87aa8b\" (UID: \"327b41fd-c8a7-4926-bc59-8869ae87aa8b\") " Nov 03 21:01:57.823018 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.822938 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/327b41fd-c8a7-4926-bc59-8869ae87aa8b-kube-api-access-mnhpg\" (UniqueName: \"kubernetes.io/projected/327b41fd-c8a7-4926-bc59-8869ae87aa8b-kube-api-access-mnhpg\") pod \"327b41fd-c8a7-4926-bc59-8869ae87aa8b\" (UID: \"327b41fd-c8a7-4926-bc59-8869ae87aa8b\") " Nov 03 21:01:57.823018 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.823005 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/327b41fd-c8a7-4926-bc59-8869ae87aa8b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/327b41fd-c8a7-4926-bc59-8869ae87aa8b-istiod-ca-cert\") pod \"327b41fd-c8a7-4926-bc59-8869ae87aa8b\" (UID: \"327b41fd-c8a7-4926-bc59-8869ae87aa8b\") " Nov 03 21:01:57.823138 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.823044 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/327b41fd-c8a7-4926-bc59-8869ae87aa8b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/327b41fd-c8a7-4926-bc59-8869ae87aa8b-istio-podinfo\") pod \"327b41fd-c8a7-4926-bc59-8869ae87aa8b\" (UID: \"327b41fd-c8a7-4926-bc59-8869ae87aa8b\") " Nov 03 21:01:57.823936 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.823359 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/327b41fd-c8a7-4926-bc59-8869ae87aa8b-workload-socket" pod "327b41fd-c8a7-4926-bc59-8869ae87aa8b" (UID: "327b41fd-c8a7-4926-bc59-8869ae87aa8b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:01:57.823936 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.823873 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/327b41fd-c8a7-4926-bc59-8869ae87aa8b-workload-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:01:57.823936 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.823896 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/327b41fd-c8a7-4926-bc59-8869ae87aa8b-credential-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:01:57.823936 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.823899 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/327b41fd-c8a7-4926-bc59-8869ae87aa8b-istiod-ca-cert" pod "327b41fd-c8a7-4926-bc59-8869ae87aa8b" (UID: "327b41fd-c8a7-4926-bc59-8869ae87aa8b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:01:57.823936 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.823913 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/327b41fd-c8a7-4926-bc59-8869ae87aa8b-workload-certs\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:01:57.823936 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.823937 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/327b41fd-c8a7-4926-bc59-8869ae87aa8b-istio-data\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:01:57.826423 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.826062 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/327b41fd-c8a7-4926-bc59-8869ae87aa8b-istio-token" pod "327b41fd-c8a7-4926-bc59-8869ae87aa8b" (UID: "327b41fd-c8a7-4926-bc59-8869ae87aa8b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:01:57.826423 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.826267 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/327b41fd-c8a7-4926-bc59-8869ae87aa8b-istio-envoy" pod "327b41fd-c8a7-4926-bc59-8869ae87aa8b" (UID: "327b41fd-c8a7-4926-bc59-8869ae87aa8b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:01:57.826423 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.826399 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/327b41fd-c8a7-4926-bc59-8869ae87aa8b-istio-podinfo" pod "327b41fd-c8a7-4926-bc59-8869ae87aa8b" (UID: "327b41fd-c8a7-4926-bc59-8869ae87aa8b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:01:57.827205 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.827181 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/327b41fd-c8a7-4926-bc59-8869ae87aa8b-kube-api-access-mnhpg" pod "327b41fd-c8a7-4926-bc59-8869ae87aa8b" (UID: "327b41fd-c8a7-4926-bc59-8869ae87aa8b"). InnerVolumeSpecName "kube-api-access-mnhpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:01:57.884145 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.884058 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b52bda-77lqz" event={"ID":"2fc27cc8-cbb7-4c99-95f7-1678fa27b72d","Type":"ContainerDied","Data":"623e82d7ec02a39df975e2c7cbc7bfa92e5739cdce86e3b1774a4a4295512c7c"} Nov 03 21:01:57.884145 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.884088 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b52bda-77lqz" Nov 03 21:01:57.884145 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.884134 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="623e82d7ec02a39df975e2c7cbc7bfa92e5739cdce86e3b1774a4a4295512c7c" Nov 03 21:01:57.886036 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.886000 4305 generic.go:356] "Generic (PLEG): container finished" podID="a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a" containerID="578e5a417d1a34d198a42091b4a70573ccb2ce9f030428a3cc36eb5db596531c" exitCode=0 Nov 03 21:01:57.886183 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.886140 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-59cjd" Nov 03 21:01:57.886246 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.886203 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-59cjd" event={"ID":"a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a","Type":"ContainerDied","Data":"578e5a417d1a34d198a42091b4a70573ccb2ce9f030428a3cc36eb5db596531c"} Nov 03 21:01:57.886310 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.886249 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-59cjd" event={"ID":"a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a","Type":"ContainerDied","Data":"04475958e0ce2c10410fa5191c8a8698eb2ad2788608f576faf940c67b1553e6"} Nov 03 21:01:57.886310 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.886278 4305 scope.go:122] "RemoveContainer" containerID="578e5a417d1a34d198a42091b4a70573ccb2ce9f030428a3cc36eb5db596531c" Nov 03 21:01:57.888582 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.888554 4305 generic.go:356] "Generic (PLEG): container finished" podID="327b41fd-c8a7-4926-bc59-8869ae87aa8b" containerID="f423dd3aac35b5995843da8d11913875dfa9f8109624326e4f6739579de731d4" exitCode=0 Nov 03 21:01:57.888677 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.888646 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-588hz" Nov 03 21:01:57.888677 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.888657 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-588hz" event={"ID":"327b41fd-c8a7-4926-bc59-8869ae87aa8b","Type":"ContainerDied","Data":"f423dd3aac35b5995843da8d11913875dfa9f8109624326e4f6739579de731d4"} Nov 03 21:01:57.888751 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.888698 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-588hz" event={"ID":"327b41fd-c8a7-4926-bc59-8869ae87aa8b","Type":"ContainerDied","Data":"afc7433a032c1b3bb7d663e9dd30c69a90d0bf860a87e62affe141b1c9d352c0"} Nov 03 21:01:57.890782 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.890760 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fad5eaf-7tg2k" event={"ID":"d4c6f39d-5dcd-49a9-967b-4303f59d75c0","Type":"ContainerStarted","Data":"753e3ed999f626f0f8d4f41f1ef6bf2cf6c09ca205110f86110bd73159959fcf"} Nov 03 21:01:57.890920 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.890803 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fad5eaf-7tg2k" event={"ID":"d4c6f39d-5dcd-49a9-967b-4303f59d75c0","Type":"ContainerStarted","Data":"76d9227f84a915154775a2b33f5cf47f33b9d12cdbde7c251d2778e9e54ca9dd"} Nov 03 21:01:57.901495 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.901094 4305 scope.go:122] "RemoveContainer" containerID="578e5a417d1a34d198a42091b4a70573ccb2ce9f030428a3cc36eb5db596531c" Nov 03 21:01:57.901748 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:01:57.901624 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"578e5a417d1a34d198a42091b4a70573ccb2ce9f030428a3cc36eb5db596531c\": container with ID starting with 578e5a417d1a34d198a42091b4a70573ccb2ce9f030428a3cc36eb5db596531c not found: ID does not exist" containerID="578e5a417d1a34d198a42091b4a70573ccb2ce9f030428a3cc36eb5db596531c" Nov 03 21:01:57.901748 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.901666 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"578e5a417d1a34d198a42091b4a70573ccb2ce9f030428a3cc36eb5db596531c"} err="failed to get container status \"578e5a417d1a34d198a42091b4a70573ccb2ce9f030428a3cc36eb5db596531c\": rpc error: code = NotFound desc = could not find container \"578e5a417d1a34d198a42091b4a70573ccb2ce9f030428a3cc36eb5db596531c\": container with ID starting with 578e5a417d1a34d198a42091b4a70573ccb2ce9f030428a3cc36eb5db596531c not found: ID does not exist" Nov 03 21:01:57.901748 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.901693 4305 scope.go:122] "RemoveContainer" containerID="f423dd3aac35b5995843da8d11913875dfa9f8109624326e4f6739579de731d4" Nov 03 21:01:57.923915 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.922597 4305 scope.go:122] "RemoveContainer" containerID="f423dd3aac35b5995843da8d11913875dfa9f8109624326e4f6739579de731d4" Nov 03 21:01:57.923915 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:01:57.922973 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f423dd3aac35b5995843da8d11913875dfa9f8109624326e4f6739579de731d4\": container with ID starting with f423dd3aac35b5995843da8d11913875dfa9f8109624326e4f6739579de731d4 not found: ID does not exist" containerID="f423dd3aac35b5995843da8d11913875dfa9f8109624326e4f6739579de731d4" Nov 03 21:01:57.923915 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.923020 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f423dd3aac35b5995843da8d11913875dfa9f8109624326e4f6739579de731d4"} err="failed to get container status \"f423dd3aac35b5995843da8d11913875dfa9f8109624326e4f6739579de731d4\": rpc error: code = NotFound desc = could not find container \"f423dd3aac35b5995843da8d11913875dfa9f8109624326e4f6739579de731d4\": container with ID starting with f423dd3aac35b5995843da8d11913875dfa9f8109624326e4f6739579de731d4 not found: ID does not exist" Nov 03 21:01:57.925277 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.925252 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/327b41fd-c8a7-4926-bc59-8869ae87aa8b-istio-envoy\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:01:57.925277 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.925278 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/327b41fd-c8a7-4926-bc59-8869ae87aa8b-istio-token\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:01:57.925759 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.925289 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnhpg\" (UniqueName: \"kubernetes.io/projected/327b41fd-c8a7-4926-bc59-8869ae87aa8b-kube-api-access-mnhpg\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:01:57.925759 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.925299 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/327b41fd-c8a7-4926-bc59-8869ae87aa8b-istiod-ca-cert\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:01:57.925759 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:57.925310 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/327b41fd-c8a7-4926-bc59-8869ae87aa8b-istio-podinfo\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:01:58.109227 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:58.109177 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fad5eaf-7tg2k" podStartSLOduration=2.109161077 podStartE2EDuration="2.109161077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:01:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:01:58.107034834 +0000 UTC m=+1353.332354629" watchObservedRunningTime="2026-11-03 21:01:58.109161077 +0000 UTC m=+1353.334480868" Nov 03 21:01:58.221529 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:58.221494 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-588hz"] Nov 03 21:01:58.228022 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:58.227995 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-588hz"] Nov 03 21:01:58.336676 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:58.336635 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-59cjd"] Nov 03 21:01:58.341410 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:58.341382 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-59cjd"] Nov 03 21:01:58.897839 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:58.897801 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="327b41fd-c8a7-4926-bc59-8869ae87aa8b" path="/var/lib/kubelet/pods/327b41fd-c8a7-4926-bc59-8869ae87aa8b/volumes" Nov 03 21:01:58.898600 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:58.898580 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a" path="/var/lib/kubelet/pods/a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a/volumes" Nov 03 21:01:58.902656 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:58.902634 4305 generic.go:356] "Generic (PLEG): container finished" podID="d4c6f39d-5dcd-49a9-967b-4303f59d75c0" containerID="753e3ed999f626f0f8d4f41f1ef6bf2cf6c09ca205110f86110bd73159959fcf" exitCode=0 Nov 03 21:01:58.902760 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:01:58.902662 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fad5eaf-7tg2k" event={"ID":"d4c6f39d-5dcd-49a9-967b-4303f59d75c0","Type":"ContainerDied","Data":"753e3ed999f626f0f8d4f41f1ef6bf2cf6c09ca205110f86110bd73159959fcf"} Nov 03 21:02:00.103496 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:00.103470 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fad5eaf-7tg2k" Nov 03 21:02:00.262679 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:00.262571 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4c6f39d-5dcd-49a9-967b-4303f59d75c0-kube-api-access-fnr5w\" (UniqueName: \"kubernetes.io/projected/d4c6f39d-5dcd-49a9-967b-4303f59d75c0-kube-api-access-fnr5w\") pod \"d4c6f39d-5dcd-49a9-967b-4303f59d75c0\" (UID: \"d4c6f39d-5dcd-49a9-967b-4303f59d75c0\") " Nov 03 21:02:00.262679 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:00.262647 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4c6f39d-5dcd-49a9-967b-4303f59d75c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4c6f39d-5dcd-49a9-967b-4303f59d75c0-maas-api-xks-ca\") pod \"d4c6f39d-5dcd-49a9-967b-4303f59d75c0\" (UID: \"d4c6f39d-5dcd-49a9-967b-4303f59d75c0\") " Nov 03 21:02:00.262936 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:00.262688 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4c6f39d-5dcd-49a9-967b-4303f59d75c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4c6f39d-5dcd-49a9-967b-4303f59d75c0-maas-api-openshift-ca\") pod \"d4c6f39d-5dcd-49a9-967b-4303f59d75c0\" (UID: \"d4c6f39d-5dcd-49a9-967b-4303f59d75c0\") " Nov 03 21:02:00.263176 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:00.263143 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4c6f39d-5dcd-49a9-967b-4303f59d75c0-maas-api-openshift-ca" pod "d4c6f39d-5dcd-49a9-967b-4303f59d75c0" (UID: "d4c6f39d-5dcd-49a9-967b-4303f59d75c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:02:00.263397 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:00.263377 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4c6f39d-5dcd-49a9-967b-4303f59d75c0-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:02:00.265698 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:00.265662 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4c6f39d-5dcd-49a9-967b-4303f59d75c0-maas-api-xks-ca" pod "d4c6f39d-5dcd-49a9-967b-4303f59d75c0" (UID: "d4c6f39d-5dcd-49a9-967b-4303f59d75c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:02:00.265834 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:00.265756 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4c6f39d-5dcd-49a9-967b-4303f59d75c0-kube-api-access-fnr5w" pod "d4c6f39d-5dcd-49a9-967b-4303f59d75c0" (UID: "d4c6f39d-5dcd-49a9-967b-4303f59d75c0"). InnerVolumeSpecName "kube-api-access-fnr5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:02:00.365368 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:00.365337 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4c6f39d-5dcd-49a9-967b-4303f59d75c0-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:02:00.365368 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:00.365366 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnr5w\" (UniqueName: \"kubernetes.io/projected/d4c6f39d-5dcd-49a9-967b-4303f59d75c0-kube-api-access-fnr5w\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:02:00.940133 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:00.932711 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fad5eaf-7tg2k" Nov 03 21:02:00.940133 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:00.932740 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fad5eaf-7tg2k" event={"ID":"d4c6f39d-5dcd-49a9-967b-4303f59d75c0","Type":"ContainerDied","Data":"76d9227f84a915154775a2b33f5cf47f33b9d12cdbde7c251d2778e9e54ca9dd"} Nov 03 21:02:00.940133 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:00.932811 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76d9227f84a915154775a2b33f5cf47f33b9d12cdbde7c251d2778e9e54ca9dd" Nov 03 21:02:01.051805 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:01.051774 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9daefaf7-f8gs7"] Nov 03 21:02:01.057634 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:01.057602 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9daefaf7-f8gs7"] Nov 03 21:02:02.896827 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:02.896796 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80e4faf1-6b07-4c82-85d5-3807a6573379" path="/var/lib/kubelet/pods/80e4faf1-6b07-4c82-85d5-3807a6573379/volumes" Nov 03 21:02:04.139094 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:04.139053 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e971f2a5-w98b8"] Nov 03 21:02:04.140663 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:04.140637 4305 cpu_manager.go:414] "containerMap: removing container" podUID="d4c6f39d-5dcd-49a9-967b-4303f59d75c0" containerName="revoke-keys" Nov 03 21:02:04.140663 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:04.140660 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4c6f39d-5dcd-49a9-967b-4303f59d75c0" containerName="revoke-keys" Nov 03 21:02:04.140845 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:04.140688 4305 cpu_manager.go:414] "containerMap: removing container" podUID="2fc27cc8-cbb7-4c99-95f7-1678fa27b72d" containerName="revoke-keys" Nov 03 21:02:04.140845 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:04.140695 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fc27cc8-cbb7-4c99-95f7-1678fa27b72d" containerName="revoke-keys" Nov 03 21:02:04.140845 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:04.140788 4305 cpu_manager.go:414] "containerMap: removing container" podUID="a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a" containerName="istio-proxy" Nov 03 21:02:04.140845 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:04.140795 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a" containerName="istio-proxy" Nov 03 21:02:04.140845 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:04.140824 4305 cpu_manager.go:414] "containerMap: removing container" podUID="327b41fd-c8a7-4926-bc59-8869ae87aa8b" containerName="istio-proxy" Nov 03 21:02:04.140845 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:04.140831 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="327b41fd-c8a7-4926-bc59-8869ae87aa8b" containerName="istio-proxy" Nov 03 21:02:04.141125 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:04.141104 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4c6f39d-5dcd-49a9-967b-4303f59d75c0" containerName="revoke-keys" Nov 03 21:02:04.141180 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:04.141128 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fc27cc8-cbb7-4c99-95f7-1678fa27b72d" containerName="revoke-keys" Nov 03 21:02:04.141180 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:04.141155 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0374ac6-461c-4e89-ac15-d8ce8dd6bb6a" containerName="istio-proxy" Nov 03 21:02:04.141180 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:04.141172 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="327b41fd-c8a7-4926-bc59-8869ae87aa8b" containerName="istio-proxy" Nov 03 21:02:04.146287 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:04.146265 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e971f2a5-w98b8" Nov 03 21:02:04.184416 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:04.184381 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5e4bf14-js862"] Nov 03 21:02:04.188882 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:04.188862 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e4bf14-js862" Nov 03 21:02:04.195351 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:04.195322 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e971f2a5-w98b8"] Nov 03 21:02:04.225239 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:04.225206 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/515e30b7-016e-4bbf-b546-66900d5f5e3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5e4bf14-js862\" (UID: \"515e30b7-016e-4bbf-b546-66900d5f5e3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e4bf14-js862" Nov 03 21:02:04.225239 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:04.225250 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k7rb\" (UniqueName: \"kubernetes.io/projected/261e3cbf-4f0e-4fc4-9e2f-11dcd6298ee5-kube-api-access-7k7rb\") pod \"maas-api-revoke-sub-e971f2a5-w98b8\" (UID: \"261e3cbf-4f0e-4fc4-9e2f-11dcd6298ee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e971f2a5-w98b8" Nov 03 21:02:04.225638 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:04.225405 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/515e30b7-016e-4bbf-b546-66900d5f5e3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5e4bf14-js862\" (UID: \"515e30b7-016e-4bbf-b546-66900d5f5e3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e4bf14-js862" Nov 03 21:02:04.225638 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:04.225585 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/261e3cbf-4f0e-4fc4-9e2f-11dcd6298ee5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e971f2a5-w98b8\" (UID: \"261e3cbf-4f0e-4fc4-9e2f-11dcd6298ee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e971f2a5-w98b8" Nov 03 21:02:04.225638 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:04.225629 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/261e3cbf-4f0e-4fc4-9e2f-11dcd6298ee5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e971f2a5-w98b8\" (UID: \"261e3cbf-4f0e-4fc4-9e2f-11dcd6298ee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e971f2a5-w98b8" Nov 03 21:02:04.225778 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:04.225707 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-688ps\" (UniqueName: \"kubernetes.io/projected/515e30b7-016e-4bbf-b546-66900d5f5e3d-kube-api-access-688ps\") pod \"maas-api-revoke-sub-a5e4bf14-js862\" (UID: \"515e30b7-016e-4bbf-b546-66900d5f5e3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e4bf14-js862" Nov 03 21:02:04.227810 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:04.227749 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:02:04.228057 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:04.227884 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:02:04.239886 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:04.239822 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77vtd\"" Nov 03 21:02:04.240169 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:04.240139 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:02:04.248195 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:04.248166 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5e4bf14-js862"] Nov 03 21:02:04.326827 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:04.326791 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/515e30b7-016e-4bbf-b546-66900d5f5e3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5e4bf14-js862\" (UID: \"515e30b7-016e-4bbf-b546-66900d5f5e3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e4bf14-js862" Nov 03 21:02:04.327037 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:04.326841 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k7rb\" (UniqueName: \"kubernetes.io/projected/261e3cbf-4f0e-4fc4-9e2f-11dcd6298ee5-kube-api-access-7k7rb\") pod \"maas-api-revoke-sub-e971f2a5-w98b8\" (UID: \"261e3cbf-4f0e-4fc4-9e2f-11dcd6298ee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e971f2a5-w98b8" Nov 03 21:02:04.327037 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:04.326911 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/515e30b7-016e-4bbf-b546-66900d5f5e3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5e4bf14-js862\" (UID: \"515e30b7-016e-4bbf-b546-66900d5f5e3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e4bf14-js862" Nov 03 21:02:04.327037 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:04.327016 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/261e3cbf-4f0e-4fc4-9e2f-11dcd6298ee5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e971f2a5-w98b8\" (UID: \"261e3cbf-4f0e-4fc4-9e2f-11dcd6298ee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e971f2a5-w98b8" Nov 03 21:02:04.327211 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:04.327054 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/261e3cbf-4f0e-4fc4-9e2f-11dcd6298ee5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e971f2a5-w98b8\" (UID: \"261e3cbf-4f0e-4fc4-9e2f-11dcd6298ee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e971f2a5-w98b8" Nov 03 21:02:04.327211 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:04.327135 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-688ps\" (UniqueName: \"kubernetes.io/projected/515e30b7-016e-4bbf-b546-66900d5f5e3d-kube-api-access-688ps\") pod \"maas-api-revoke-sub-a5e4bf14-js862\" (UID: \"515e30b7-016e-4bbf-b546-66900d5f5e3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e4bf14-js862" Nov 03 21:02:04.327626 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:04.327600 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/515e30b7-016e-4bbf-b546-66900d5f5e3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5e4bf14-js862\" (UID: \"515e30b7-016e-4bbf-b546-66900d5f5e3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e4bf14-js862" Nov 03 21:02:04.327940 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:04.327875 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/261e3cbf-4f0e-4fc4-9e2f-11dcd6298ee5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e971f2a5-w98b8\" (UID: \"261e3cbf-4f0e-4fc4-9e2f-11dcd6298ee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e971f2a5-w98b8" Nov 03 21:02:04.329717 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:04.329695 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/515e30b7-016e-4bbf-b546-66900d5f5e3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5e4bf14-js862\" (UID: \"515e30b7-016e-4bbf-b546-66900d5f5e3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e4bf14-js862" Nov 03 21:02:04.329796 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:04.329755 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/261e3cbf-4f0e-4fc4-9e2f-11dcd6298ee5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e971f2a5-w98b8\" (UID: \"261e3cbf-4f0e-4fc4-9e2f-11dcd6298ee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e971f2a5-w98b8" Nov 03 21:02:04.388118 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:04.388086 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k7rb\" (UniqueName: \"kubernetes.io/projected/261e3cbf-4f0e-4fc4-9e2f-11dcd6298ee5-kube-api-access-7k7rb\") pod \"maas-api-revoke-sub-e971f2a5-w98b8\" (UID: \"261e3cbf-4f0e-4fc4-9e2f-11dcd6298ee5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e971f2a5-w98b8" Nov 03 21:02:04.388286 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:04.388086 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-688ps\" (UniqueName: \"kubernetes.io/projected/515e30b7-016e-4bbf-b546-66900d5f5e3d-kube-api-access-688ps\") pod \"maas-api-revoke-sub-a5e4bf14-js862\" (UID: \"515e30b7-016e-4bbf-b546-66900d5f5e3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e4bf14-js862" Nov 03 21:02:04.460582 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:04.460550 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e971f2a5-w98b8" Nov 03 21:02:04.507184 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:04.507146 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e4bf14-js862" Nov 03 21:02:04.853104 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:04.853067 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e971f2a5-w98b8"] Nov 03 21:02:04.853324 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:02:04.853299 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod261e3cbf_4f0e_4fc4_9e2f_11dcd6298ee5.slice/crio-bbee559d36bd0991636a27416d9b53b2447ccdcdb46d24c379a36d9c08dd56ca WatchSource:0}: Error finding container bbee559d36bd0991636a27416d9b53b2447ccdcdb46d24c379a36d9c08dd56ca: Status 404 returned error can't find the container with id bbee559d36bd0991636a27416d9b53b2447ccdcdb46d24c379a36d9c08dd56ca Nov 03 21:02:04.914575 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:04.914547 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5e4bf14-js862"] Nov 03 21:02:04.914705 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:02:04.914657 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod515e30b7_016e_4bbf_b546_66900d5f5e3d.slice/crio-0ef24425d7ff22e28eda3a655799695154965805e1335c94a689f32e3523d95a WatchSource:0}: Error finding container 0ef24425d7ff22e28eda3a655799695154965805e1335c94a689f32e3523d95a: Status 404 returned error can't find the container with id 0ef24425d7ff22e28eda3a655799695154965805e1335c94a689f32e3523d95a Nov 03 21:02:04.967506 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:04.967469 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e971f2a5-w98b8" event={"ID":"261e3cbf-4f0e-4fc4-9e2f-11dcd6298ee5","Type":"ContainerStarted","Data":"bbee559d36bd0991636a27416d9b53b2447ccdcdb46d24c379a36d9c08dd56ca"} Nov 03 21:02:04.969115 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:04.969084 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e4bf14-js862" event={"ID":"515e30b7-016e-4bbf-b546-66900d5f5e3d","Type":"ContainerStarted","Data":"0ef24425d7ff22e28eda3a655799695154965805e1335c94a689f32e3523d95a"} Nov 03 21:02:05.979176 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:05.979149 4305 generic.go:356] "Generic (PLEG): container finished" podID="261e3cbf-4f0e-4fc4-9e2f-11dcd6298ee5" containerID="55383d33fd02610b7c73156a75c26bf7a11e1eca4424338ab8bf9f652fbc6116" exitCode=0 Nov 03 21:02:05.979589 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:05.979298 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e971f2a5-w98b8" event={"ID":"261e3cbf-4f0e-4fc4-9e2f-11dcd6298ee5","Type":"ContainerDied","Data":"55383d33fd02610b7c73156a75c26bf7a11e1eca4424338ab8bf9f652fbc6116"} Nov 03 21:02:05.981169 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:05.981128 4305 generic.go:356] "Generic (PLEG): container finished" podID="515e30b7-016e-4bbf-b546-66900d5f5e3d" containerID="e68d0b79d6f99eeaee550444c8a88915d3c3f18fd152d092f72d44c02220107a" exitCode=0 Nov 03 21:02:05.981287 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:05.981178 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e4bf14-js862" event={"ID":"515e30b7-016e-4bbf-b546-66900d5f5e3d","Type":"ContainerDied","Data":"e68d0b79d6f99eeaee550444c8a88915d3c3f18fd152d092f72d44c02220107a"} Nov 03 21:02:06.091833 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:06.091737 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b5f75e9-7k9qt"] Nov 03 21:02:06.100392 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:06.100363 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b5f75e9-7k9qt"] Nov 03 21:02:06.174584 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:06.174551 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:02:07.042925 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:07.042887 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68525bc3-6784-42a7-9b5b-6c1fe167d454" path="/var/lib/kubelet/pods/68525bc3-6784-42a7-9b5b-6c1fe167d454/volumes" Nov 03 21:02:08.255576 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:08.255540 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e4bf14-js862" Nov 03 21:02:08.347062 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:08.339122 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/515e30b7-016e-4bbf-b546-66900d5f5e3d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/515e30b7-016e-4bbf-b546-66900d5f5e3d-maas-api-openshift-ca\") pod \"515e30b7-016e-4bbf-b546-66900d5f5e3d\" (UID: \"515e30b7-016e-4bbf-b546-66900d5f5e3d\") " Nov 03 21:02:08.347062 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:08.339207 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/515e30b7-016e-4bbf-b546-66900d5f5e3d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/515e30b7-016e-4bbf-b546-66900d5f5e3d-maas-api-xks-ca\") pod \"515e30b7-016e-4bbf-b546-66900d5f5e3d\" (UID: \"515e30b7-016e-4bbf-b546-66900d5f5e3d\") " Nov 03 21:02:08.347062 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:08.339272 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/515e30b7-016e-4bbf-b546-66900d5f5e3d-kube-api-access-688ps\" (UniqueName: \"kubernetes.io/projected/515e30b7-016e-4bbf-b546-66900d5f5e3d-kube-api-access-688ps\") pod \"515e30b7-016e-4bbf-b546-66900d5f5e3d\" (UID: \"515e30b7-016e-4bbf-b546-66900d5f5e3d\") " Nov 03 21:02:08.347062 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:08.345626 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e971f2a5-w98b8" Nov 03 21:02:08.376781 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:08.374172 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/515e30b7-016e-4bbf-b546-66900d5f5e3d-kube-api-access-688ps" pod "515e30b7-016e-4bbf-b546-66900d5f5e3d" (UID: "515e30b7-016e-4bbf-b546-66900d5f5e3d"). InnerVolumeSpecName "kube-api-access-688ps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:02:08.376781 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:08.374546 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/515e30b7-016e-4bbf-b546-66900d5f5e3d-maas-api-openshift-ca" pod "515e30b7-016e-4bbf-b546-66900d5f5e3d" (UID: "515e30b7-016e-4bbf-b546-66900d5f5e3d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:02:08.404460 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:08.403975 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/515e30b7-016e-4bbf-b546-66900d5f5e3d-maas-api-xks-ca" pod "515e30b7-016e-4bbf-b546-66900d5f5e3d" (UID: "515e30b7-016e-4bbf-b546-66900d5f5e3d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:02:08.446460 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:08.445869 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e4bf14-js862" event={"ID":"515e30b7-016e-4bbf-b546-66900d5f5e3d","Type":"ContainerDied","Data":"0ef24425d7ff22e28eda3a655799695154965805e1335c94a689f32e3523d95a"} Nov 03 21:02:08.446460 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:08.445935 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ef24425d7ff22e28eda3a655799695154965805e1335c94a689f32e3523d95a" Nov 03 21:02:08.446460 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:08.446020 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e4bf14-js862" Nov 03 21:02:08.446792 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:08.446610 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/261e3cbf-4f0e-4fc4-9e2f-11dcd6298ee5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/261e3cbf-4f0e-4fc4-9e2f-11dcd6298ee5-maas-api-openshift-ca\") pod \"261e3cbf-4f0e-4fc4-9e2f-11dcd6298ee5\" (UID: \"261e3cbf-4f0e-4fc4-9e2f-11dcd6298ee5\") " Nov 03 21:02:08.455461 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:08.446846 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/261e3cbf-4f0e-4fc4-9e2f-11dcd6298ee5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/261e3cbf-4f0e-4fc4-9e2f-11dcd6298ee5-maas-api-xks-ca\") pod \"261e3cbf-4f0e-4fc4-9e2f-11dcd6298ee5\" (UID: \"261e3cbf-4f0e-4fc4-9e2f-11dcd6298ee5\") " Nov 03 21:02:08.455461 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:08.447050 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/261e3cbf-4f0e-4fc4-9e2f-11dcd6298ee5-kube-api-access-7k7rb\" (UniqueName: \"kubernetes.io/projected/261e3cbf-4f0e-4fc4-9e2f-11dcd6298ee5-kube-api-access-7k7rb\") pod \"261e3cbf-4f0e-4fc4-9e2f-11dcd6298ee5\" (UID: \"261e3cbf-4f0e-4fc4-9e2f-11dcd6298ee5\") " Nov 03 21:02:08.455461 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:08.447671 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/515e30b7-016e-4bbf-b546-66900d5f5e3d-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:02:08.455461 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:08.447694 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/515e30b7-016e-4bbf-b546-66900d5f5e3d-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:02:08.455461 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:08.447710 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-688ps\" (UniqueName: \"kubernetes.io/projected/515e30b7-016e-4bbf-b546-66900d5f5e3d-kube-api-access-688ps\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:02:08.509558 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:08.509499 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/261e3cbf-4f0e-4fc4-9e2f-11dcd6298ee5-maas-api-openshift-ca" pod "261e3cbf-4f0e-4fc4-9e2f-11dcd6298ee5" (UID: "261e3cbf-4f0e-4fc4-9e2f-11dcd6298ee5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:02:08.510601 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:08.510286 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e971f2a5-w98b8" event={"ID":"261e3cbf-4f0e-4fc4-9e2f-11dcd6298ee5","Type":"ContainerDied","Data":"bbee559d36bd0991636a27416d9b53b2447ccdcdb46d24c379a36d9c08dd56ca"} Nov 03 21:02:08.510601 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:08.510345 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bbee559d36bd0991636a27416d9b53b2447ccdcdb46d24c379a36d9c08dd56ca" Nov 03 21:02:08.510601 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:08.510412 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e971f2a5-w98b8" Nov 03 21:02:08.550768 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:08.550679 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/261e3cbf-4f0e-4fc4-9e2f-11dcd6298ee5-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:02:08.889495 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:08.888550 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" podUID="8ee1f028-8503-4034-9954-eb06fcf402e7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:08.911051 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:08.910913 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/261e3cbf-4f0e-4fc4-9e2f-11dcd6298ee5-kube-api-access-7k7rb" pod "261e3cbf-4f0e-4fc4-9e2f-11dcd6298ee5" (UID: "261e3cbf-4f0e-4fc4-9e2f-11dcd6298ee5"). InnerVolumeSpecName "kube-api-access-7k7rb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:02:08.925624 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:08.925560 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/261e3cbf-4f0e-4fc4-9e2f-11dcd6298ee5-maas-api-xks-ca" pod "261e3cbf-4f0e-4fc4-9e2f-11dcd6298ee5" (UID: "261e3cbf-4f0e-4fc4-9e2f-11dcd6298ee5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:02:09.002836 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:09.002726 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" podUID="7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 03 21:02:09.021949 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:09.021872 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7k7rb\" (UniqueName: \"kubernetes.io/projected/261e3cbf-4f0e-4fc4-9e2f-11dcd6298ee5-kube-api-access-7k7rb\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:02:09.021949 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:09.021915 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/261e3cbf-4f0e-4fc4-9e2f-11dcd6298ee5-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:02:09.073784 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:09.072574 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" podUID="d5d3dcaa-a51a-40c6-8049-871b300552e1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded" Nov 03 21:02:09.079203 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:09.078386 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" podUID="7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:09.104411 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:09.103530 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" podUID="63a704b4-51ce-46c3-8007-34b308aff332" containerName="main" probeResult="failure" output="Get \"https://10.134.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:09.104742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:09.104711 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" podUID="be06bc96-808a-46e6-9872-01baf1a11a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:09.105244 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:09.105153 4305 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" podUID="793b8ca3-3a2e-49a1-9be4-4971ba59df09" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": context deadline exceeded" Nov 03 21:02:09.126717 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:09.126678 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" podUID="006c5eae-06af-447b-a1d8-cd719cce2a92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Nov 03 21:02:09.305731 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:09.305616 4305 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" podUID="6e7f30a9-0862-48a5-9ab7-e692e75bb294" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:09.356811 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:09.356752 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" podUID="8793f095-97b1-4e0a-8382-b541f90a0de2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 03 21:02:09.378476 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:09.377628 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" podUID="80de7944-70f6-49e1-a0e8-4b82d5caad0d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:09.413578 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:09.413531 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" podUID="793b8ca3-3a2e-49a1-9be4-4971ba59df09" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 03 21:02:09.477915 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:09.477876 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" podUID="6e7f30a9-0862-48a5-9ab7-e692e75bb294" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 03 21:02:09.542796 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:09.542747 4305 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" podUID="be06bc96-808a-46e6-9872-01baf1a11a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.107:8000/health\": context deadline exceeded" Nov 03 21:02:09.899665 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:09.899591 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" podUID="8ee1f028-8503-4034-9954-eb06fcf402e7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:09.982463 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:09.910648 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" Nov 03 21:02:10.021997 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:10.021948 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" podUID="7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:10.030000 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:10.029964 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" Nov 03 21:02:10.116583 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:10.116534 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" podUID="63a704b4-51ce-46c3-8007-34b308aff332" containerName="main" probeResult="failure" output="Get \"https://10.134.0.101:8000/ready\": context deadline exceeded" Nov 03 21:02:10.117247 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:10.117215 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" podUID="d5d3dcaa-a51a-40c6-8049-871b300552e1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded" Nov 03 21:02:10.117759 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:10.117738 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" Nov 03 21:02:10.117954 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:10.117940 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" Nov 03 21:02:10.242178 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:10.222606 4305 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" podUID="7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:10.242178 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:10.222753 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" podUID="7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 03 21:02:10.245599 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:10.245556 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" podUID="be06bc96-808a-46e6-9872-01baf1a11a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.107:8000/ready\": context deadline exceeded" Nov 03 21:02:10.245946 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:10.245924 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" Nov 03 21:02:10.249794 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:10.249773 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" Nov 03 21:02:10.314915 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:10.301642 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" podUID="006c5eae-06af-447b-a1d8-cd719cce2a92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:10.350463 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:10.324559 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" Nov 03 21:02:10.378651 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:10.360542 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" podUID="8793f095-97b1-4e0a-8382-b541f90a0de2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:10.378651 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:10.360981 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" Nov 03 21:02:10.399478 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:10.379041 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" podUID="80de7944-70f6-49e1-a0e8-4b82d5caad0d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:10.399478 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:10.379413 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" Nov 03 21:02:10.430416 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:10.430366 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" podUID="793b8ca3-3a2e-49a1-9be4-4971ba59df09" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 03 21:02:10.431083 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:10.431061 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" Nov 03 21:02:10.479460 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:10.478758 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" podUID="6e7f30a9-0862-48a5-9ab7-e692e75bb294" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:10.479460 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:10.479117 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" Nov 03 21:02:10.978680 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:10.978641 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" podUID="8ee1f028-8503-4034-9954-eb06fcf402e7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": context deadline exceeded" Nov 03 21:02:11.046507 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:11.027060 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" podUID="7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 03 21:02:11.163465 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:11.160263 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" podUID="d5d3dcaa-a51a-40c6-8049-871b300552e1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:11.163465 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:11.160637 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" podUID="63a704b4-51ce-46c3-8007-34b308aff332" containerName="main" probeResult="failure" output="Get \"https://10.134.0.101:8000/ready\": context deadline exceeded" Nov 03 21:02:11.230527 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:11.224097 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" podUID="7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:11.261970 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:11.261571 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" podUID="be06bc96-808a-46e6-9872-01baf1a11a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:11.322552 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:11.322095 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" podUID="006c5eae-06af-447b-a1d8-cd719cce2a92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:11.376688 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:11.371480 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" podUID="8793f095-97b1-4e0a-8382-b541f90a0de2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 03 21:02:11.503528 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:11.455088 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" podUID="793b8ca3-3a2e-49a1-9be4-4971ba59df09" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 03 21:02:11.505732 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:11.487142 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" podUID="6e7f30a9-0862-48a5-9ab7-e692e75bb294" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:11.551500 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:11.389880 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" podUID="80de7944-70f6-49e1-a0e8-4b82d5caad0d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:12.092460 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:12.047161 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" podUID="8ee1f028-8503-4034-9954-eb06fcf402e7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:12.147517 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:12.080567 4305 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" podUID="006c5eae-06af-447b-a1d8-cd719cce2a92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": context deadline exceeded" Nov 03 21:02:12.147817 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:12.089424 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" podUID="7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:12.207540 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:12.199844 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" podUID="63a704b4-51ce-46c3-8007-34b308aff332" containerName="main" probeResult="failure" output="Get \"https://10.134.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:12.207540 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:12.201099 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" podUID="d5d3dcaa-a51a-40c6-8049-871b300552e1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded" Nov 03 21:02:12.277373 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:12.268269 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" podUID="7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 03 21:02:12.277373 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:12.273789 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" podUID="be06bc96-808a-46e6-9872-01baf1a11a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:12.362311 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:12.348522 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" podUID="006c5eae-06af-447b-a1d8-cd719cce2a92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:12.466461 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:12.438098 4305 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" podUID="d5d3dcaa-a51a-40c6-8049-871b300552e1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:12.466830 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:12.466793 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" podUID="8793f095-97b1-4e0a-8382-b541f90a0de2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:12.561460 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:12.561116 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" podUID="6e7f30a9-0862-48a5-9ab7-e692e75bb294" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:12.565457 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:12.561724 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" podUID="793b8ca3-3a2e-49a1-9be4-4971ba59df09" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:12.735854 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:12.729019 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" podUID="80de7944-70f6-49e1-a0e8-4b82d5caad0d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:13.052513 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:13.049936 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" Nov 03 21:02:13.052513 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:13.050059 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" Nov 03 21:02:13.052513 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:13.050092 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" Nov 03 21:02:13.052513 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:13.050131 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" Nov 03 21:02:13.052513 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:13.050168 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" Nov 03 21:02:13.052513 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:13.050203 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" Nov 03 21:02:13.052513 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:13.050235 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" Nov 03 21:02:13.052513 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:13.050264 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" Nov 03 21:02:13.052513 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:13.050293 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" Nov 03 21:02:13.052513 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:13.050323 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" Nov 03 21:02:13.065518 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:13.065487 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" Nov 03 21:02:14.929265 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:14.929221 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c1d3bc7-mfc85"] Nov 03 21:02:14.929936 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:14.929921 4305 cpu_manager.go:414] "containerMap: removing container" podUID="261e3cbf-4f0e-4fc4-9e2f-11dcd6298ee5" containerName="revoke-keys" Nov 03 21:02:14.929981 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:14.929938 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="261e3cbf-4f0e-4fc4-9e2f-11dcd6298ee5" containerName="revoke-keys" Nov 03 21:02:14.929981 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:14.929957 4305 cpu_manager.go:414] "containerMap: removing container" podUID="515e30b7-016e-4bbf-b546-66900d5f5e3d" containerName="revoke-keys" Nov 03 21:02:14.929981 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:14.929963 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="515e30b7-016e-4bbf-b546-66900d5f5e3d" containerName="revoke-keys" Nov 03 21:02:14.930145 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:14.930134 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="261e3cbf-4f0e-4fc4-9e2f-11dcd6298ee5" containerName="revoke-keys" Nov 03 21:02:14.930181 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:14.930148 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="515e30b7-016e-4bbf-b546-66900d5f5e3d" containerName="revoke-keys" Nov 03 21:02:14.933769 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:14.933751 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1d3bc7-mfc85" Nov 03 21:02:14.989774 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:14.989740 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:02:14.989774 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:14.989740 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:02:14.990042 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:14.989904 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:02:14.996015 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:14.995985 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c1d3bc7-mfc85"] Nov 03 21:02:15.008227 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:15.008181 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b8cdfb4-1311-4d1c-8b40-64ce920f0248-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c1d3bc7-mfc85\" (UID: \"2b8cdfb4-1311-4d1c-8b40-64ce920f0248\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1d3bc7-mfc85" Nov 03 21:02:15.008462 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:15.008265 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wh68\" (UniqueName: \"kubernetes.io/projected/2b8cdfb4-1311-4d1c-8b40-64ce920f0248-kube-api-access-5wh68\") pod \"maas-api-revoke-sub-4c1d3bc7-mfc85\" (UID: \"2b8cdfb4-1311-4d1c-8b40-64ce920f0248\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1d3bc7-mfc85" Nov 03 21:02:15.008462 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:15.008394 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b8cdfb4-1311-4d1c-8b40-64ce920f0248-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c1d3bc7-mfc85\" (UID: \"2b8cdfb4-1311-4d1c-8b40-64ce920f0248\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1d3bc7-mfc85" Nov 03 21:02:15.036091 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:15.036061 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77vtd\"" Nov 03 21:02:15.110350 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:15.110309 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b8cdfb4-1311-4d1c-8b40-64ce920f0248-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c1d3bc7-mfc85\" (UID: \"2b8cdfb4-1311-4d1c-8b40-64ce920f0248\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1d3bc7-mfc85" Nov 03 21:02:15.110600 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:15.110393 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wh68\" (UniqueName: \"kubernetes.io/projected/2b8cdfb4-1311-4d1c-8b40-64ce920f0248-kube-api-access-5wh68\") pod \"maas-api-revoke-sub-4c1d3bc7-mfc85\" (UID: \"2b8cdfb4-1311-4d1c-8b40-64ce920f0248\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1d3bc7-mfc85" Nov 03 21:02:15.110600 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:15.110427 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b8cdfb4-1311-4d1c-8b40-64ce920f0248-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c1d3bc7-mfc85\" (UID: \"2b8cdfb4-1311-4d1c-8b40-64ce920f0248\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1d3bc7-mfc85" Nov 03 21:02:15.111121 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:15.111095 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b8cdfb4-1311-4d1c-8b40-64ce920f0248-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c1d3bc7-mfc85\" (UID: \"2b8cdfb4-1311-4d1c-8b40-64ce920f0248\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1d3bc7-mfc85" Nov 03 21:02:15.113288 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:15.113260 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b8cdfb4-1311-4d1c-8b40-64ce920f0248-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c1d3bc7-mfc85\" (UID: \"2b8cdfb4-1311-4d1c-8b40-64ce920f0248\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1d3bc7-mfc85" Nov 03 21:02:15.171861 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:15.171829 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wh68\" (UniqueName: \"kubernetes.io/projected/2b8cdfb4-1311-4d1c-8b40-64ce920f0248-kube-api-access-5wh68\") pod \"maas-api-revoke-sub-4c1d3bc7-mfc85\" (UID: \"2b8cdfb4-1311-4d1c-8b40-64ce920f0248\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1d3bc7-mfc85" Nov 03 21:02:15.255628 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:15.255543 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1d3bc7-mfc85" Nov 03 21:02:15.645918 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:15.645883 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c1d3bc7-mfc85"] Nov 03 21:02:15.653199 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:02:15.653161 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b8cdfb4_1311_4d1c_8b40_64ce920f0248.slice/crio-6d2a18b7f1639c1feb5dcfdcb7edd91963dd0745c2506ef04dd66604b25e5666 WatchSource:0}: Error finding container 6d2a18b7f1639c1feb5dcfdcb7edd91963dd0745c2506ef04dd66604b25e5666: Status 404 returned error can't find the container with id 6d2a18b7f1639c1feb5dcfdcb7edd91963dd0745c2506ef04dd66604b25e5666 Nov 03 21:02:15.866074 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:15.866037 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1d3bc7-mfc85" event={"ID":"2b8cdfb4-1311-4d1c-8b40-64ce920f0248","Type":"ContainerStarted","Data":"6d2a18b7f1639c1feb5dcfdcb7edd91963dd0745c2506ef04dd66604b25e5666"} Nov 03 21:02:15.884057 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:15.884023 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0af8368d-btdft"] Nov 03 21:02:15.889775 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:15.889748 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af8368d-btdft" Nov 03 21:02:15.946075 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:15.946044 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0af8368d-btdft"] Nov 03 21:02:16.024422 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:16.024363 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3fa728b-bd28-42b6-98de-9853c0a04f15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0af8368d-btdft\" (UID: \"d3fa728b-bd28-42b6-98de-9853c0a04f15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af8368d-btdft" Nov 03 21:02:16.024667 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:16.024502 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3fa728b-bd28-42b6-98de-9853c0a04f15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0af8368d-btdft\" (UID: \"d3fa728b-bd28-42b6-98de-9853c0a04f15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af8368d-btdft" Nov 03 21:02:16.024667 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:16.024631 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6m8v\" (UniqueName: \"kubernetes.io/projected/d3fa728b-bd28-42b6-98de-9853c0a04f15-kube-api-access-b6m8v\") pod \"maas-api-revoke-sub-0af8368d-btdft\" (UID: \"d3fa728b-bd28-42b6-98de-9853c0a04f15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af8368d-btdft" Nov 03 21:02:16.126522 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:16.126492 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3fa728b-bd28-42b6-98de-9853c0a04f15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0af8368d-btdft\" (UID: \"d3fa728b-bd28-42b6-98de-9853c0a04f15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af8368d-btdft" Nov 03 21:02:16.126717 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:16.126538 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3fa728b-bd28-42b6-98de-9853c0a04f15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0af8368d-btdft\" (UID: \"d3fa728b-bd28-42b6-98de-9853c0a04f15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af8368d-btdft" Nov 03 21:02:16.126717 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:16.126590 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6m8v\" (UniqueName: \"kubernetes.io/projected/d3fa728b-bd28-42b6-98de-9853c0a04f15-kube-api-access-b6m8v\") pod \"maas-api-revoke-sub-0af8368d-btdft\" (UID: \"d3fa728b-bd28-42b6-98de-9853c0a04f15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af8368d-btdft" Nov 03 21:02:16.127244 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:16.127221 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3fa728b-bd28-42b6-98de-9853c0a04f15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0af8368d-btdft\" (UID: \"d3fa728b-bd28-42b6-98de-9853c0a04f15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af8368d-btdft" Nov 03 21:02:16.129186 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:16.129164 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3fa728b-bd28-42b6-98de-9853c0a04f15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0af8368d-btdft\" (UID: \"d3fa728b-bd28-42b6-98de-9853c0a04f15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af8368d-btdft" Nov 03 21:02:16.187529 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:16.187501 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6m8v\" (UniqueName: \"kubernetes.io/projected/d3fa728b-bd28-42b6-98de-9853c0a04f15-kube-api-access-b6m8v\") pod \"maas-api-revoke-sub-0af8368d-btdft\" (UID: \"d3fa728b-bd28-42b6-98de-9853c0a04f15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af8368d-btdft" Nov 03 21:02:16.204479 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:16.204417 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af8368d-btdft" Nov 03 21:02:16.357511 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:02:16.357420 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:02:16.681905 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:02:16.681867 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3fa728b_bd28_42b6_98de_9853c0a04f15.slice/crio-92b2d33fb7b62340b6d5e6bb9bb0dba64b6bd89dad90c695dc4823e38fa89857 WatchSource:0}: Error finding container 92b2d33fb7b62340b6d5e6bb9bb0dba64b6bd89dad90c695dc4823e38fa89857: Status 404 returned error can't find the container with id 92b2d33fb7b62340b6d5e6bb9bb0dba64b6bd89dad90c695dc4823e38fa89857 Nov 03 21:02:16.682766 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:16.682734 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0af8368d-btdft"] Nov 03 21:02:16.878234 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:16.878202 4305 generic.go:356] "Generic (PLEG): container finished" podID="2b8cdfb4-1311-4d1c-8b40-64ce920f0248" containerID="13c25083621c3ab3656bb562ebc0c9d3223d53988839c19668692f73b94e7248" exitCode=0 Nov 03 21:02:16.878411 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:16.878339 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1d3bc7-mfc85" event={"ID":"2b8cdfb4-1311-4d1c-8b40-64ce920f0248","Type":"ContainerDied","Data":"13c25083621c3ab3656bb562ebc0c9d3223d53988839c19668692f73b94e7248"} Nov 03 21:02:16.879826 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:16.879802 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af8368d-btdft" event={"ID":"d3fa728b-bd28-42b6-98de-9853c0a04f15","Type":"ContainerStarted","Data":"92b2d33fb7b62340b6d5e6bb9bb0dba64b6bd89dad90c695dc4823e38fa89857"} Nov 03 21:02:17.891904 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:17.891873 4305 generic.go:356] "Generic (PLEG): container finished" podID="d3fa728b-bd28-42b6-98de-9853c0a04f15" containerID="69279a13c80d80e4c6bed49347dd4d876bc6b6e3c7ab55b2ddf43baea8545e5f" exitCode=0 Nov 03 21:02:17.892352 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:17.891916 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af8368d-btdft" event={"ID":"d3fa728b-bd28-42b6-98de-9853c0a04f15","Type":"ContainerDied","Data":"69279a13c80d80e4c6bed49347dd4d876bc6b6e3c7ab55b2ddf43baea8545e5f"} Nov 03 21:02:18.070028 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:18.069997 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1d3bc7-mfc85" Nov 03 21:02:18.083603 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:18.083572 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b32989ae-4hzzn"] Nov 03 21:02:18.088126 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:18.088101 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b32989ae-4hzzn"] Nov 03 21:02:18.158852 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:18.158821 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b8cdfb4-1311-4d1c-8b40-64ce920f0248-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b8cdfb4-1311-4d1c-8b40-64ce920f0248-maas-api-xks-ca\") pod \"2b8cdfb4-1311-4d1c-8b40-64ce920f0248\" (UID: \"2b8cdfb4-1311-4d1c-8b40-64ce920f0248\") " Nov 03 21:02:18.159053 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:18.158953 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b8cdfb4-1311-4d1c-8b40-64ce920f0248-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b8cdfb4-1311-4d1c-8b40-64ce920f0248-maas-api-openshift-ca\") pod \"2b8cdfb4-1311-4d1c-8b40-64ce920f0248\" (UID: \"2b8cdfb4-1311-4d1c-8b40-64ce920f0248\") " Nov 03 21:02:18.159053 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:18.159006 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b8cdfb4-1311-4d1c-8b40-64ce920f0248-kube-api-access-5wh68\" (UniqueName: \"kubernetes.io/projected/2b8cdfb4-1311-4d1c-8b40-64ce920f0248-kube-api-access-5wh68\") pod \"2b8cdfb4-1311-4d1c-8b40-64ce920f0248\" (UID: \"2b8cdfb4-1311-4d1c-8b40-64ce920f0248\") " Nov 03 21:02:18.159376 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:18.159351 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b8cdfb4-1311-4d1c-8b40-64ce920f0248-maas-api-openshift-ca" pod "2b8cdfb4-1311-4d1c-8b40-64ce920f0248" (UID: "2b8cdfb4-1311-4d1c-8b40-64ce920f0248"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:02:18.159656 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:18.159640 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b8cdfb4-1311-4d1c-8b40-64ce920f0248-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:02:18.161301 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:18.161244 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b8cdfb4-1311-4d1c-8b40-64ce920f0248-maas-api-xks-ca" pod "2b8cdfb4-1311-4d1c-8b40-64ce920f0248" (UID: "2b8cdfb4-1311-4d1c-8b40-64ce920f0248"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:02:18.161512 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:18.161487 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b8cdfb4-1311-4d1c-8b40-64ce920f0248-kube-api-access-5wh68" pod "2b8cdfb4-1311-4d1c-8b40-64ce920f0248" (UID: "2b8cdfb4-1311-4d1c-8b40-64ce920f0248"). InnerVolumeSpecName "kube-api-access-5wh68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:02:18.261728 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:18.261692 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b8cdfb4-1311-4d1c-8b40-64ce920f0248-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:02:18.261728 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:18.261726 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wh68\" (UniqueName: \"kubernetes.io/projected/2b8cdfb4-1311-4d1c-8b40-64ce920f0248-kube-api-access-5wh68\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:02:18.899237 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:18.899198 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbcc1750-05a6-41a4-b0ea-0390b5ba7d77" path="/var/lib/kubelet/pods/bbcc1750-05a6-41a4-b0ea-0390b5ba7d77/volumes" Nov 03 21:02:18.903578 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:18.903554 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1d3bc7-mfc85" Nov 03 21:02:18.903578 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:18.903535 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c1d3bc7-mfc85" event={"ID":"2b8cdfb4-1311-4d1c-8b40-64ce920f0248","Type":"ContainerDied","Data":"6d2a18b7f1639c1feb5dcfdcb7edd91963dd0745c2506ef04dd66604b25e5666"} Nov 03 21:02:18.903890 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:18.903671 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d2a18b7f1639c1feb5dcfdcb7edd91963dd0745c2506ef04dd66604b25e5666" Nov 03 21:02:19.101208 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:19.101171 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af8368d-btdft" Nov 03 21:02:19.173466 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:19.173357 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3fa728b-bd28-42b6-98de-9853c0a04f15-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3fa728b-bd28-42b6-98de-9853c0a04f15-maas-api-xks-ca\") pod \"d3fa728b-bd28-42b6-98de-9853c0a04f15\" (UID: \"d3fa728b-bd28-42b6-98de-9853c0a04f15\") " Nov 03 21:02:19.173675 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:19.173658 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3fa728b-bd28-42b6-98de-9853c0a04f15-kube-api-access-b6m8v\" (UniqueName: \"kubernetes.io/projected/d3fa728b-bd28-42b6-98de-9853c0a04f15-kube-api-access-b6m8v\") pod \"d3fa728b-bd28-42b6-98de-9853c0a04f15\" (UID: \"d3fa728b-bd28-42b6-98de-9853c0a04f15\") " Nov 03 21:02:19.173773 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:19.173709 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3fa728b-bd28-42b6-98de-9853c0a04f15-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3fa728b-bd28-42b6-98de-9853c0a04f15-maas-api-openshift-ca\") pod \"d3fa728b-bd28-42b6-98de-9853c0a04f15\" (UID: \"d3fa728b-bd28-42b6-98de-9853c0a04f15\") " Nov 03 21:02:19.174099 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:19.174070 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3fa728b-bd28-42b6-98de-9853c0a04f15-maas-api-openshift-ca" pod "d3fa728b-bd28-42b6-98de-9853c0a04f15" (UID: "d3fa728b-bd28-42b6-98de-9853c0a04f15"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:02:19.174387 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:19.174367 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3fa728b-bd28-42b6-98de-9853c0a04f15-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:02:19.176061 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:19.176039 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3fa728b-bd28-42b6-98de-9853c0a04f15-maas-api-xks-ca" pod "d3fa728b-bd28-42b6-98de-9853c0a04f15" (UID: "d3fa728b-bd28-42b6-98de-9853c0a04f15"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:02:19.176130 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:19.176057 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3fa728b-bd28-42b6-98de-9853c0a04f15-kube-api-access-b6m8v" pod "d3fa728b-bd28-42b6-98de-9853c0a04f15" (UID: "d3fa728b-bd28-42b6-98de-9853c0a04f15"). InnerVolumeSpecName "kube-api-access-b6m8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:02:19.276088 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:19.276047 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6m8v\" (UniqueName: \"kubernetes.io/projected/d3fa728b-bd28-42b6-98de-9853c0a04f15-kube-api-access-b6m8v\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:02:19.276088 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:19.276085 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3fa728b-bd28-42b6-98de-9853c0a04f15-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:02:19.914630 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:19.914604 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af8368d-btdft" Nov 03 21:02:19.915033 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:19.914638 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0af8368d-btdft" event={"ID":"d3fa728b-bd28-42b6-98de-9853c0a04f15","Type":"ContainerDied","Data":"92b2d33fb7b62340b6d5e6bb9bb0dba64b6bd89dad90c695dc4823e38fa89857"} Nov 03 21:02:19.915033 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:19.914685 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92b2d33fb7b62340b6d5e6bb9bb0dba64b6bd89dad90c695dc4823e38fa89857" Nov 03 21:02:21.570194 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:21.570158 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j"] Nov 03 21:02:21.570920 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:21.570902 4305 cpu_manager.go:414] "containerMap: removing container" podUID="2b8cdfb4-1311-4d1c-8b40-64ce920f0248" containerName="revoke-keys" Nov 03 21:02:21.570985 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:21.570921 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b8cdfb4-1311-4d1c-8b40-64ce920f0248" containerName="revoke-keys" Nov 03 21:02:21.570985 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:21.570941 4305 cpu_manager.go:414] "containerMap: removing container" podUID="d3fa728b-bd28-42b6-98de-9853c0a04f15" containerName="revoke-keys" Nov 03 21:02:21.570985 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:21.570946 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3fa728b-bd28-42b6-98de-9853c0a04f15" containerName="revoke-keys" Nov 03 21:02:21.571139 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:21.571050 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b8cdfb4-1311-4d1c-8b40-64ce920f0248" containerName="revoke-keys" Nov 03 21:02:21.571139 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:21.571063 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3fa728b-bd28-42b6-98de-9853c0a04f15" containerName="revoke-keys" Nov 03 21:02:21.574940 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:21.574917 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j" Nov 03 21:02:21.716373 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:21.716328 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2366aada-a95b-404a-b5a7-454a1670c687-workload-certs\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j\" (UID: \"2366aada-a95b-404a-b5a7-454a1670c687\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j" Nov 03 21:02:21.716625 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:21.716390 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2366aada-a95b-404a-b5a7-454a1670c687-credential-socket\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j\" (UID: \"2366aada-a95b-404a-b5a7-454a1670c687\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j" Nov 03 21:02:21.716625 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:21.716589 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z4kr\" (UniqueName: \"kubernetes.io/projected/2366aada-a95b-404a-b5a7-454a1670c687-kube-api-access-4z4kr\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j\" (UID: \"2366aada-a95b-404a-b5a7-454a1670c687\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j" Nov 03 21:02:21.716761 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:21.716646 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2366aada-a95b-404a-b5a7-454a1670c687-istio-data\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j\" (UID: \"2366aada-a95b-404a-b5a7-454a1670c687\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j" Nov 03 21:02:21.716761 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:21.716674 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2366aada-a95b-404a-b5a7-454a1670c687-istio-token\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j\" (UID: \"2366aada-a95b-404a-b5a7-454a1670c687\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j" Nov 03 21:02:21.716870 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:21.716853 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2366aada-a95b-404a-b5a7-454a1670c687-workload-socket\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j\" (UID: \"2366aada-a95b-404a-b5a7-454a1670c687\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j" Nov 03 21:02:21.716931 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:21.716887 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2366aada-a95b-404a-b5a7-454a1670c687-istio-podinfo\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j\" (UID: \"2366aada-a95b-404a-b5a7-454a1670c687\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j" Nov 03 21:02:21.716995 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:21.716972 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2366aada-a95b-404a-b5a7-454a1670c687-istiod-ca-cert\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j\" (UID: \"2366aada-a95b-404a-b5a7-454a1670c687\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j" Nov 03 21:02:21.717035 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:21.717021 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2366aada-a95b-404a-b5a7-454a1670c687-istio-envoy\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j\" (UID: \"2366aada-a95b-404a-b5a7-454a1670c687\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j" Nov 03 21:02:21.818527 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:21.818487 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z4kr\" (UniqueName: \"kubernetes.io/projected/2366aada-a95b-404a-b5a7-454a1670c687-kube-api-access-4z4kr\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j\" (UID: \"2366aada-a95b-404a-b5a7-454a1670c687\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j" Nov 03 21:02:21.818737 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:21.818544 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2366aada-a95b-404a-b5a7-454a1670c687-istio-data\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j\" (UID: \"2366aada-a95b-404a-b5a7-454a1670c687\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j" Nov 03 21:02:21.818737 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:21.818569 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2366aada-a95b-404a-b5a7-454a1670c687-istio-token\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j\" (UID: \"2366aada-a95b-404a-b5a7-454a1670c687\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j" Nov 03 21:02:21.818737 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:21.818695 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2366aada-a95b-404a-b5a7-454a1670c687-workload-socket\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j\" (UID: \"2366aada-a95b-404a-b5a7-454a1670c687\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j" Nov 03 21:02:21.818737 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:21.818720 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2366aada-a95b-404a-b5a7-454a1670c687-istio-podinfo\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j\" (UID: \"2366aada-a95b-404a-b5a7-454a1670c687\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j" Nov 03 21:02:21.818976 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:21.818753 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2366aada-a95b-404a-b5a7-454a1670c687-istiod-ca-cert\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j\" (UID: \"2366aada-a95b-404a-b5a7-454a1670c687\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j" Nov 03 21:02:21.818976 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:21.818785 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2366aada-a95b-404a-b5a7-454a1670c687-istio-envoy\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j\" (UID: \"2366aada-a95b-404a-b5a7-454a1670c687\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j" Nov 03 21:02:21.819083 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:21.818985 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2366aada-a95b-404a-b5a7-454a1670c687-istio-data\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j\" (UID: \"2366aada-a95b-404a-b5a7-454a1670c687\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j" Nov 03 21:02:21.819083 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:21.819075 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2366aada-a95b-404a-b5a7-454a1670c687-workload-certs\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j\" (UID: \"2366aada-a95b-404a-b5a7-454a1670c687\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j" Nov 03 21:02:21.819192 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:21.819108 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2366aada-a95b-404a-b5a7-454a1670c687-credential-socket\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j\" (UID: \"2366aada-a95b-404a-b5a7-454a1670c687\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j" Nov 03 21:02:21.819251 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:21.819230 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2366aada-a95b-404a-b5a7-454a1670c687-workload-socket\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j\" (UID: \"2366aada-a95b-404a-b5a7-454a1670c687\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j" Nov 03 21:02:21.819811 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:21.819782 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2366aada-a95b-404a-b5a7-454a1670c687-istiod-ca-cert\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j\" (UID: \"2366aada-a95b-404a-b5a7-454a1670c687\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j" Nov 03 21:02:21.819986 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:21.819945 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j"] Nov 03 21:02:21.820075 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:21.820059 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2366aada-a95b-404a-b5a7-454a1670c687-workload-certs\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j\" (UID: \"2366aada-a95b-404a-b5a7-454a1670c687\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j" Nov 03 21:02:21.820140 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:21.820121 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2366aada-a95b-404a-b5a7-454a1670c687-credential-socket\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j\" (UID: \"2366aada-a95b-404a-b5a7-454a1670c687\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j" Nov 03 21:02:21.821826 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:21.821763 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2366aada-a95b-404a-b5a7-454a1670c687-istio-podinfo\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j\" (UID: \"2366aada-a95b-404a-b5a7-454a1670c687\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j" Nov 03 21:02:21.822568 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:21.822545 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2366aada-a95b-404a-b5a7-454a1670c687-istio-envoy\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j\" (UID: \"2366aada-a95b-404a-b5a7-454a1670c687\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j" Nov 03 21:02:21.880057 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:21.880027 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2366aada-a95b-404a-b5a7-454a1670c687-istio-token\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j\" (UID: \"2366aada-a95b-404a-b5a7-454a1670c687\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j" Nov 03 21:02:21.880224 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:21.880081 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z4kr\" (UniqueName: \"kubernetes.io/projected/2366aada-a95b-404a-b5a7-454a1670c687-kube-api-access-4z4kr\") pod \"e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j\" (UID: \"2366aada-a95b-404a-b5a7-454a1670c687\") " pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j" Nov 03 21:02:21.900074 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:21.900038 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j" Nov 03 21:02:22.060525 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:22.060492 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9029abd-84sxf"] Nov 03 21:02:22.064074 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:22.064041 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9029abd-84sxf"] Nov 03 21:02:22.354031 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:02:22.353997 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2366aada_a95b_404a_b5a7_454a1670c687.slice/crio-bc86ac6cdaa9cf246dcd9a323992a7701c4a7b6fe8cf804f39f4ffb98d36448e WatchSource:0}: Error finding container bc86ac6cdaa9cf246dcd9a323992a7701c4a7b6fe8cf804f39f4ffb98d36448e: Status 404 returned error can't find the container with id bc86ac6cdaa9cf246dcd9a323992a7701c4a7b6fe8cf804f39f4ffb98d36448e Nov 03 21:02:22.355404 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:22.355378 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j"] Nov 03 21:02:22.357112 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:22.357073 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:02:22.357210 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:22.357160 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:02:22.357252 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:22.357204 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:02:22.898414 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:22.898369 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11edcf66-04da-4a99-a3d8-67635dd2ff0b" path="/var/lib/kubelet/pods/11edcf66-04da-4a99-a3d8-67635dd2ff0b/volumes" Nov 03 21:02:22.945931 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:22.945775 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j" event={"ID":"2366aada-a95b-404a-b5a7-454a1670c687","Type":"ContainerStarted","Data":"0486deee2a7aa8f3c903d1dc0ebf3fd12299e62bcc6f18296e98eb9b34718bf2"} Nov 03 21:02:22.945931 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:22.945842 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j" event={"ID":"2366aada-a95b-404a-b5a7-454a1670c687","Type":"ContainerStarted","Data":"bc86ac6cdaa9cf246dcd9a323992a7701c4a7b6fe8cf804f39f4ffb98d36448e"} Nov 03 21:02:23.119026 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:23.118966 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j" podStartSLOduration=2.118946583 podStartE2EDuration="2.118946583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:02:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:02:23.115792379 +0000 UTC m=+1378.341112194" watchObservedRunningTime="2026-11-03 21:02:23.118946583 +0000 UTC m=+1378.344266375" Nov 03 21:02:23.900959 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:23.900927 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j" Nov 03 21:02:23.903116 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:23.903091 4305 patch_prober.go:29] interesting pod/e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 03 21:02:23.903259 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:23.903153 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j" podUID="2366aada-a95b-404a-b5a7-454a1670c687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 03 21:02:24.900508 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:24.900477 4305 patch_prober.go:29] interesting pod/e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 03 21:02:24.900690 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:24.900525 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j" podUID="2366aada-a95b-404a-b5a7-454a1670c687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 03 21:02:25.900948 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:25.900915 4305 patch_prober.go:29] interesting pod/e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 03 21:02:25.901388 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:25.900975 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j" podUID="2366aada-a95b-404a-b5a7-454a1670c687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 03 21:02:26.751259 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:26.751225 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-661f63e8-h52c9"] Nov 03 21:02:26.779275 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:26.779244 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-661f63e8-h52c9" Nov 03 21:02:26.813358 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:26.813322 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-661f63e8-h52c9"] Nov 03 21:02:26.832090 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:26.832047 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a81f4383-a20c-4c43-824a-2987ff0c8f38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-661f63e8-h52c9\" (UID: \"a81f4383-a20c-4c43-824a-2987ff0c8f38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-661f63e8-h52c9" Nov 03 21:02:26.832311 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:26.832243 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a81f4383-a20c-4c43-824a-2987ff0c8f38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-661f63e8-h52c9\" (UID: \"a81f4383-a20c-4c43-824a-2987ff0c8f38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-661f63e8-h52c9" Nov 03 21:02:26.832421 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:26.832391 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fchmn\" (UniqueName: \"kubernetes.io/projected/a81f4383-a20c-4c43-824a-2987ff0c8f38-kube-api-access-fchmn\") pod \"maas-api-revoke-sub-661f63e8-h52c9\" (UID: \"a81f4383-a20c-4c43-824a-2987ff0c8f38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-661f63e8-h52c9" Nov 03 21:02:26.878097 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:26.877840 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:02:26.878097 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:26.877966 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:02:26.878665 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:26.878639 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77vtd\"" Nov 03 21:02:26.878890 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:26.878854 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:02:26.900931 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:26.900902 4305 patch_prober.go:29] interesting pod/e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 03 21:02:26.901090 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:26.900952 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j" podUID="2366aada-a95b-404a-b5a7-454a1670c687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 03 21:02:26.934102 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:26.934058 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fchmn\" (UniqueName: \"kubernetes.io/projected/a81f4383-a20c-4c43-824a-2987ff0c8f38-kube-api-access-fchmn\") pod \"maas-api-revoke-sub-661f63e8-h52c9\" (UID: \"a81f4383-a20c-4c43-824a-2987ff0c8f38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-661f63e8-h52c9" Nov 03 21:02:26.934306 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:26.934176 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a81f4383-a20c-4c43-824a-2987ff0c8f38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-661f63e8-h52c9\" (UID: \"a81f4383-a20c-4c43-824a-2987ff0c8f38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-661f63e8-h52c9" Nov 03 21:02:26.934306 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:26.934300 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a81f4383-a20c-4c43-824a-2987ff0c8f38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-661f63e8-h52c9\" (UID: \"a81f4383-a20c-4c43-824a-2987ff0c8f38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-661f63e8-h52c9" Nov 03 21:02:26.934975 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:26.934953 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a81f4383-a20c-4c43-824a-2987ff0c8f38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-661f63e8-h52c9\" (UID: \"a81f4383-a20c-4c43-824a-2987ff0c8f38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-661f63e8-h52c9" Nov 03 21:02:26.936948 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:26.936927 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a81f4383-a20c-4c43-824a-2987ff0c8f38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-661f63e8-h52c9\" (UID: \"a81f4383-a20c-4c43-824a-2987ff0c8f38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-661f63e8-h52c9" Nov 03 21:02:26.995979 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:26.995949 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fchmn\" (UniqueName: \"kubernetes.io/projected/a81f4383-a20c-4c43-824a-2987ff0c8f38-kube-api-access-fchmn\") pod \"maas-api-revoke-sub-661f63e8-h52c9\" (UID: \"a81f4383-a20c-4c43-824a-2987ff0c8f38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-661f63e8-h52c9" Nov 03 21:02:27.095544 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:27.095412 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-661f63e8-h52c9" Nov 03 21:02:27.510051 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:02:27.510015 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda81f4383_a20c_4c43_824a_2987ff0c8f38.slice/crio-a8f0254c102b774f53eb522202e7d39fbcc6cdf5f327e1db7aae6d3f4ed8bb35 WatchSource:0}: Error finding container a8f0254c102b774f53eb522202e7d39fbcc6cdf5f327e1db7aae6d3f4ed8bb35: Status 404 returned error can't find the container with id a8f0254c102b774f53eb522202e7d39fbcc6cdf5f327e1db7aae6d3f4ed8bb35 Nov 03 21:02:27.510904 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:27.510882 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-661f63e8-h52c9"] Nov 03 21:02:27.901019 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:27.900986 4305 patch_prober.go:29] interesting pod/e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 03 21:02:27.901188 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:27.901047 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j" podUID="2366aada-a95b-404a-b5a7-454a1670c687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 03 21:02:28.024031 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:28.023925 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-661f63e8-h52c9" event={"ID":"a81f4383-a20c-4c43-824a-2987ff0c8f38","Type":"ContainerStarted","Data":"63b7bd9c85ae7ec4a51da08ed59af05a60277e7f2350fbdc587cfc2fbc903b7f"} Nov 03 21:02:28.024031 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:28.023997 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-661f63e8-h52c9" event={"ID":"a81f4383-a20c-4c43-824a-2987ff0c8f38","Type":"ContainerStarted","Data":"a8f0254c102b774f53eb522202e7d39fbcc6cdf5f327e1db7aae6d3f4ed8bb35"} Nov 03 21:02:28.146463 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:28.146402 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-661f63e8-h52c9" podStartSLOduration=2.146383621 podStartE2EDuration="2.146383621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:02:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:02:28.141866608 +0000 UTC m=+1383.367186401" watchObservedRunningTime="2026-11-03 21:02:28.146383621 +0000 UTC m=+1383.371703452" Nov 03 21:02:28.221797 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:28.221754 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4"] Nov 03 21:02:28.229015 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:28.228510 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4" Nov 03 21:02:28.295931 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:28.295883 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-46590df2-gw-openshift-default-dockercfg-pbhlj\"" Nov 03 21:02:28.423818 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:28.411720 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ecb5391-cd65-4987-8f36-95ac0f5371dd-istio-data\") pod \"e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4\" (UID: \"8ecb5391-cd65-4987-8f36-95ac0f5371dd\") " pod="openshift-ingress/e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4" Nov 03 21:02:28.445985 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:28.445944 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmlqh\" (UniqueName: \"kubernetes.io/projected/8ecb5391-cd65-4987-8f36-95ac0f5371dd-kube-api-access-mmlqh\") pod \"e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4\" (UID: \"8ecb5391-cd65-4987-8f36-95ac0f5371dd\") " pod="openshift-ingress/e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4" Nov 03 21:02:28.446406 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:28.446379 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ecb5391-cd65-4987-8f36-95ac0f5371dd-istio-podinfo\") pod \"e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4\" (UID: \"8ecb5391-cd65-4987-8f36-95ac0f5371dd\") " pod="openshift-ingress/e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4" Nov 03 21:02:28.446765 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:28.446688 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ecb5391-cd65-4987-8f36-95ac0f5371dd-workload-socket\") pod \"e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4\" (UID: \"8ecb5391-cd65-4987-8f36-95ac0f5371dd\") " pod="openshift-ingress/e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4" Nov 03 21:02:28.446994 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:28.446930 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ecb5391-cd65-4987-8f36-95ac0f5371dd-istio-token\") pod \"e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4\" (UID: \"8ecb5391-cd65-4987-8f36-95ac0f5371dd\") " pod="openshift-ingress/e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4" Nov 03 21:02:28.447225 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:28.447172 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ecb5391-cd65-4987-8f36-95ac0f5371dd-istiod-ca-cert\") pod \"e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4\" (UID: \"8ecb5391-cd65-4987-8f36-95ac0f5371dd\") " pod="openshift-ingress/e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4" Nov 03 21:02:28.447389 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:28.447378 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ecb5391-cd65-4987-8f36-95ac0f5371dd-credential-socket\") pod \"e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4\" (UID: \"8ecb5391-cd65-4987-8f36-95ac0f5371dd\") " pod="openshift-ingress/e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4" Nov 03 21:02:28.447693 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:28.447680 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ecb5391-cd65-4987-8f36-95ac0f5371dd-workload-certs\") pod \"e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4\" (UID: \"8ecb5391-cd65-4987-8f36-95ac0f5371dd\") " pod="openshift-ingress/e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4" Nov 03 21:02:28.447874 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:28.447859 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ecb5391-cd65-4987-8f36-95ac0f5371dd-istio-envoy\") pod \"e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4\" (UID: \"8ecb5391-cd65-4987-8f36-95ac0f5371dd\") " pod="openshift-ingress/e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4" Nov 03 21:02:28.450926 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:28.450902 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4"] Nov 03 21:02:28.551411 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:28.549620 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ecb5391-cd65-4987-8f36-95ac0f5371dd-istio-podinfo\") pod \"e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4\" (UID: \"8ecb5391-cd65-4987-8f36-95ac0f5371dd\") " pod="openshift-ingress/e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4" Nov 03 21:02:28.551411 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:28.549698 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ecb5391-cd65-4987-8f36-95ac0f5371dd-workload-socket\") pod \"e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4\" (UID: \"8ecb5391-cd65-4987-8f36-95ac0f5371dd\") " pod="openshift-ingress/e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4" Nov 03 21:02:28.551411 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:28.549736 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ecb5391-cd65-4987-8f36-95ac0f5371dd-istio-token\") pod \"e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4\" (UID: \"8ecb5391-cd65-4987-8f36-95ac0f5371dd\") " pod="openshift-ingress/e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4" Nov 03 21:02:28.551411 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:28.549807 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ecb5391-cd65-4987-8f36-95ac0f5371dd-istiod-ca-cert\") pod \"e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4\" (UID: \"8ecb5391-cd65-4987-8f36-95ac0f5371dd\") " pod="openshift-ingress/e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4" Nov 03 21:02:28.551411 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:28.549856 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ecb5391-cd65-4987-8f36-95ac0f5371dd-credential-socket\") pod \"e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4\" (UID: \"8ecb5391-cd65-4987-8f36-95ac0f5371dd\") " pod="openshift-ingress/e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4" Nov 03 21:02:28.551411 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:28.549979 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ecb5391-cd65-4987-8f36-95ac0f5371dd-workload-certs\") pod \"e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4\" (UID: \"8ecb5391-cd65-4987-8f36-95ac0f5371dd\") " pod="openshift-ingress/e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4" Nov 03 21:02:28.551411 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:28.550022 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ecb5391-cd65-4987-8f36-95ac0f5371dd-istio-envoy\") pod \"e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4\" (UID: \"8ecb5391-cd65-4987-8f36-95ac0f5371dd\") " pod="openshift-ingress/e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4" Nov 03 21:02:28.551411 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:28.550201 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ecb5391-cd65-4987-8f36-95ac0f5371dd-istio-data\") pod \"e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4\" (UID: \"8ecb5391-cd65-4987-8f36-95ac0f5371dd\") " pod="openshift-ingress/e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4" Nov 03 21:02:28.551411 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:28.550242 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmlqh\" (UniqueName: \"kubernetes.io/projected/8ecb5391-cd65-4987-8f36-95ac0f5371dd-kube-api-access-mmlqh\") pod \"e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4\" (UID: \"8ecb5391-cd65-4987-8f36-95ac0f5371dd\") " pod="openshift-ingress/e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4" Nov 03 21:02:28.551411 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:28.550406 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ecb5391-cd65-4987-8f36-95ac0f5371dd-workload-socket\") pod \"e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4\" (UID: \"8ecb5391-cd65-4987-8f36-95ac0f5371dd\") " pod="openshift-ingress/e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4" Nov 03 21:02:28.551411 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:28.550599 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ecb5391-cd65-4987-8f36-95ac0f5371dd-workload-certs\") pod \"e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4\" (UID: \"8ecb5391-cd65-4987-8f36-95ac0f5371dd\") " pod="openshift-ingress/e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4" Nov 03 21:02:28.551411 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:28.550943 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ecb5391-cd65-4987-8f36-95ac0f5371dd-istio-data\") pod \"e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4\" (UID: \"8ecb5391-cd65-4987-8f36-95ac0f5371dd\") " pod="openshift-ingress/e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4" Nov 03 21:02:28.551411 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:28.551313 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ecb5391-cd65-4987-8f36-95ac0f5371dd-istiod-ca-cert\") pod \"e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4\" (UID: \"8ecb5391-cd65-4987-8f36-95ac0f5371dd\") " pod="openshift-ingress/e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4" Nov 03 21:02:28.552257 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:28.551739 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ecb5391-cd65-4987-8f36-95ac0f5371dd-credential-socket\") pod \"e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4\" (UID: \"8ecb5391-cd65-4987-8f36-95ac0f5371dd\") " pod="openshift-ingress/e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4" Nov 03 21:02:28.628507 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:28.620107 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ecb5391-cd65-4987-8f36-95ac0f5371dd-istio-envoy\") pod \"e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4\" (UID: \"8ecb5391-cd65-4987-8f36-95ac0f5371dd\") " pod="openshift-ingress/e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4" Nov 03 21:02:28.629560 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:28.629498 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ecb5391-cd65-4987-8f36-95ac0f5371dd-istio-podinfo\") pod \"e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4\" (UID: \"8ecb5391-cd65-4987-8f36-95ac0f5371dd\") " pod="openshift-ingress/e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4" Nov 03 21:02:28.630221 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:28.630140 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ecb5391-cd65-4987-8f36-95ac0f5371dd-istio-token\") pod \"e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4\" (UID: \"8ecb5391-cd65-4987-8f36-95ac0f5371dd\") " pod="openshift-ingress/e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4" Nov 03 21:02:28.637262 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:28.637222 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmlqh\" (UniqueName: \"kubernetes.io/projected/8ecb5391-cd65-4987-8f36-95ac0f5371dd-kube-api-access-mmlqh\") pod \"e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4\" (UID: \"8ecb5391-cd65-4987-8f36-95ac0f5371dd\") " pod="openshift-ingress/e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4" Nov 03 21:02:28.846694 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:28.846661 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4" Nov 03 21:02:28.900933 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:28.900896 4305 patch_prober.go:29] interesting pod/e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Nov 03 21:02:28.901146 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:28.900966 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j" podUID="2366aada-a95b-404a-b5a7-454a1670c687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Nov 03 21:02:29.051131 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:29.051092 4305 generic.go:356] "Generic (PLEG): container finished" podID="a81f4383-a20c-4c43-824a-2987ff0c8f38" containerID="63b7bd9c85ae7ec4a51da08ed59af05a60277e7f2350fbdc587cfc2fbc903b7f" exitCode=0 Nov 03 21:02:29.051694 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:29.051319 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-661f63e8-h52c9" event={"ID":"a81f4383-a20c-4c43-824a-2987ff0c8f38","Type":"ContainerDied","Data":"63b7bd9c85ae7ec4a51da08ed59af05a60277e7f2350fbdc587cfc2fbc903b7f"} Nov 03 21:02:29.220886 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:29.220805 4305 scope.go:122] "RemoveContainer" containerID="d9814b638bf2433354ec201446b0e221cba4994f028d9c4a448b177d0622bb02" Nov 03 21:02:29.237620 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:29.237597 4305 scope.go:122] "RemoveContainer" containerID="0b5f22ea14aa96b3eaf56c9f3ec7a14cf95cfef50b7c58a4ccb6a8735e9fad96" Nov 03 21:02:29.253969 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:29.253938 4305 scope.go:122] "RemoveContainer" containerID="7e428e941394b9fbdf1802809df5237008d3f74dc5c299a0ffe26dc185bd093d" Nov 03 21:02:29.269635 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:29.269606 4305 scope.go:122] "RemoveContainer" containerID="285fcb1b85714e7ef017771157e327968cff1d0b9cfbfff97bb4b38e92ddb58a" Nov 03 21:02:29.290048 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:29.285898 4305 scope.go:122] "RemoveContainer" containerID="95430a1363f89f9b8392141dba66bfcc0e0db818b4ec518d8e812c596e3d7df1" Nov 03 21:02:29.299862 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:29.299825 4305 scope.go:122] "RemoveContainer" containerID="3611f92a517f15afb6c596e185871b41ab8061bab24a9ad3399537f241abf346" Nov 03 21:02:29.314305 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:29.314276 4305 scope.go:122] "RemoveContainer" containerID="b633fbda66c098af94074ddacecd8b846671760567a1e73b4855582b12453a23" Nov 03 21:02:29.329036 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:29.329015 4305 scope.go:122] "RemoveContainer" containerID="398a0ff7840008cd05ecead6644e9fb12c6161160a0629c6ae6a4a4dce152680" Nov 03 21:02:29.336581 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:29.336550 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4"] Nov 03 21:02:29.337490 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:02:29.337461 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ecb5391_cd65_4987_8f36_95ac0f5371dd.slice/crio-85f843b2fda36fc0855ec040be26b6650ff3742aa54fb7534a1ab474c22f033b WatchSource:0}: Error finding container 85f843b2fda36fc0855ec040be26b6650ff3742aa54fb7534a1ab474c22f033b: Status 404 returned error can't find the container with id 85f843b2fda36fc0855ec040be26b6650ff3742aa54fb7534a1ab474c22f033b Nov 03 21:02:29.344217 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:29.344111 4305 scope.go:122] "RemoveContainer" containerID="3828920ba73658dcc620bca4fae8d2dabb522f981dc05db247e94cd1ea233791" Nov 03 21:02:29.344600 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:29.344563 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:02:29.344693 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:29.344659 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:02:29.344765 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:29.344707 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:02:29.377483 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:29.377455 4305 scope.go:122] "RemoveContainer" containerID="589020f0d49141260b3eb465465fe871c6429598454b16571789e29ea4426d89" Nov 03 21:02:30.067245 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:30.067195 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4" event={"ID":"8ecb5391-cd65-4987-8f36-95ac0f5371dd","Type":"ContainerStarted","Data":"53264c14ee737b1b67b6bf8dea0f1dda0bdc2fd15096e018205a2ec05dc024b4"} Nov 03 21:02:30.067824 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:30.067268 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4" event={"ID":"8ecb5391-cd65-4987-8f36-95ac0f5371dd","Type":"ContainerStarted","Data":"85f843b2fda36fc0855ec040be26b6650ff3742aa54fb7534a1ab474c22f033b"} Nov 03 21:02:30.226155 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:30.226088 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4" podStartSLOduration=2.226063657 podStartE2EDuration="2.226063657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:02:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:02:30.213818113 +0000 UTC m=+1385.439137946" watchObservedRunningTime="2026-11-03 21:02:30.226063657 +0000 UTC m=+1385.451383448" Nov 03 21:02:30.308263 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:30.308231 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-661f63e8-h52c9" Nov 03 21:02:30.392042 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:30.392005 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a81f4383-a20c-4c43-824a-2987ff0c8f38-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a81f4383-a20c-4c43-824a-2987ff0c8f38-maas-api-openshift-ca\") pod \"a81f4383-a20c-4c43-824a-2987ff0c8f38\" (UID: \"a81f4383-a20c-4c43-824a-2987ff0c8f38\") " Nov 03 21:02:30.392232 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:30.392117 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a81f4383-a20c-4c43-824a-2987ff0c8f38-kube-api-access-fchmn\" (UniqueName: \"kubernetes.io/projected/a81f4383-a20c-4c43-824a-2987ff0c8f38-kube-api-access-fchmn\") pod \"a81f4383-a20c-4c43-824a-2987ff0c8f38\" (UID: \"a81f4383-a20c-4c43-824a-2987ff0c8f38\") " Nov 03 21:02:30.392319 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:30.392304 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a81f4383-a20c-4c43-824a-2987ff0c8f38-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a81f4383-a20c-4c43-824a-2987ff0c8f38-maas-api-xks-ca\") pod \"a81f4383-a20c-4c43-824a-2987ff0c8f38\" (UID: \"a81f4383-a20c-4c43-824a-2987ff0c8f38\") " Nov 03 21:02:30.392528 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:30.392500 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a81f4383-a20c-4c43-824a-2987ff0c8f38-maas-api-openshift-ca" pod "a81f4383-a20c-4c43-824a-2987ff0c8f38" (UID: "a81f4383-a20c-4c43-824a-2987ff0c8f38"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:02:30.393081 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:30.393056 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a81f4383-a20c-4c43-824a-2987ff0c8f38-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:02:30.395535 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:30.395499 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a81f4383-a20c-4c43-824a-2987ff0c8f38-maas-api-xks-ca" pod "a81f4383-a20c-4c43-824a-2987ff0c8f38" (UID: "a81f4383-a20c-4c43-824a-2987ff0c8f38"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:02:30.413104 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:30.413062 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a81f4383-a20c-4c43-824a-2987ff0c8f38-kube-api-access-fchmn" pod "a81f4383-a20c-4c43-824a-2987ff0c8f38" (UID: "a81f4383-a20c-4c43-824a-2987ff0c8f38"). InnerVolumeSpecName "kube-api-access-fchmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:02:30.504530 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:30.504486 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fchmn\" (UniqueName: \"kubernetes.io/projected/a81f4383-a20c-4c43-824a-2987ff0c8f38-kube-api-access-fchmn\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:02:30.504530 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:30.504531 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a81f4383-a20c-4c43-824a-2987ff0c8f38-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:02:30.591453 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:30.591241 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j" Nov 03 21:02:30.593475 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:30.591891 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j" Nov 03 21:02:30.593475 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:30.592951 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j" Nov 03 21:02:30.637102 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:30.637069 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cf9b10d-ktw77"] Nov 03 21:02:30.637806 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:30.637790 4305 cpu_manager.go:414] "containerMap: removing container" podUID="a81f4383-a20c-4c43-824a-2987ff0c8f38" containerName="revoke-keys" Nov 03 21:02:30.637859 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:30.637807 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a81f4383-a20c-4c43-824a-2987ff0c8f38" containerName="revoke-keys" Nov 03 21:02:30.637961 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:30.637950 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="a81f4383-a20c-4c43-824a-2987ff0c8f38" containerName="revoke-keys" Nov 03 21:02:30.641966 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:30.641938 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf9b10d-ktw77" Nov 03 21:02:30.707394 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:30.707356 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fa000fe-eafe-4adb-a4a7-28251831502d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5cf9b10d-ktw77\" (UID: \"2fa000fe-eafe-4adb-a4a7-28251831502d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf9b10d-ktw77" Nov 03 21:02:30.707394 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:30.707403 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8h2p\" (UniqueName: \"kubernetes.io/projected/2fa000fe-eafe-4adb-a4a7-28251831502d-kube-api-access-x8h2p\") pod \"maas-api-revoke-sub-5cf9b10d-ktw77\" (UID: \"2fa000fe-eafe-4adb-a4a7-28251831502d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf9b10d-ktw77" Nov 03 21:02:30.707837 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:30.707803 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fa000fe-eafe-4adb-a4a7-28251831502d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5cf9b10d-ktw77\" (UID: \"2fa000fe-eafe-4adb-a4a7-28251831502d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf9b10d-ktw77" Nov 03 21:02:30.718534 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:30.718499 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cf9b10d-ktw77"] Nov 03 21:02:30.809826 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:30.809782 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fa000fe-eafe-4adb-a4a7-28251831502d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5cf9b10d-ktw77\" (UID: \"2fa000fe-eafe-4adb-a4a7-28251831502d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf9b10d-ktw77" Nov 03 21:02:30.810044 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:30.809836 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8h2p\" (UniqueName: \"kubernetes.io/projected/2fa000fe-eafe-4adb-a4a7-28251831502d-kube-api-access-x8h2p\") pod \"maas-api-revoke-sub-5cf9b10d-ktw77\" (UID: \"2fa000fe-eafe-4adb-a4a7-28251831502d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf9b10d-ktw77" Nov 03 21:02:30.810111 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:30.810067 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fa000fe-eafe-4adb-a4a7-28251831502d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5cf9b10d-ktw77\" (UID: \"2fa000fe-eafe-4adb-a4a7-28251831502d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf9b10d-ktw77" Nov 03 21:02:30.810650 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:30.810627 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fa000fe-eafe-4adb-a4a7-28251831502d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5cf9b10d-ktw77\" (UID: \"2fa000fe-eafe-4adb-a4a7-28251831502d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf9b10d-ktw77" Nov 03 21:02:30.812665 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:30.812643 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fa000fe-eafe-4adb-a4a7-28251831502d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5cf9b10d-ktw77\" (UID: \"2fa000fe-eafe-4adb-a4a7-28251831502d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf9b10d-ktw77" Nov 03 21:02:30.847290 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:30.847248 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4" Nov 03 21:02:30.853143 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:30.853113 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4" Nov 03 21:02:30.902365 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:30.902316 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8h2p\" (UniqueName: \"kubernetes.io/projected/2fa000fe-eafe-4adb-a4a7-28251831502d-kube-api-access-x8h2p\") pod \"maas-api-revoke-sub-5cf9b10d-ktw77\" (UID: \"2fa000fe-eafe-4adb-a4a7-28251831502d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf9b10d-ktw77" Nov 03 21:02:30.961041 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:30.961004 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf9b10d-ktw77" Nov 03 21:02:31.082799 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:31.082773 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-661f63e8-h52c9" Nov 03 21:02:31.083520 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:31.082795 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-661f63e8-h52c9" event={"ID":"a81f4383-a20c-4c43-824a-2987ff0c8f38","Type":"ContainerDied","Data":"a8f0254c102b774f53eb522202e7d39fbcc6cdf5f327e1db7aae6d3f4ed8bb35"} Nov 03 21:02:31.083520 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:31.083490 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8f0254c102b774f53eb522202e7d39fbcc6cdf5f327e1db7aae6d3f4ed8bb35" Nov 03 21:02:31.087700 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:31.087675 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4" Nov 03 21:02:31.089534 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:31.089507 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4" Nov 03 21:02:31.382592 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:02:31.382562 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fa000fe_eafe_4adb_a4a7_28251831502d.slice/crio-4ae7a7ed57e9bfd618269de5784d043355c9dc4ddb388cebb792431ccee1cd27 WatchSource:0}: Error finding container 4ae7a7ed57e9bfd618269de5784d043355c9dc4ddb388cebb792431ccee1cd27: Status 404 returned error can't find the container with id 4ae7a7ed57e9bfd618269de5784d043355c9dc4ddb388cebb792431ccee1cd27 Nov 03 21:02:31.384243 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:31.384219 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cf9b10d-ktw77"] Nov 03 21:02:32.098134 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:32.098026 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf9b10d-ktw77" event={"ID":"2fa000fe-eafe-4adb-a4a7-28251831502d","Type":"ContainerStarted","Data":"b1f5fcca462a110a31f89fdad970f1c3c4cd83f7ed7e0e477fbabc4903226b93"} Nov 03 21:02:32.098134 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:32.098087 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf9b10d-ktw77" event={"ID":"2fa000fe-eafe-4adb-a4a7-28251831502d","Type":"ContainerStarted","Data":"4ae7a7ed57e9bfd618269de5784d043355c9dc4ddb388cebb792431ccee1cd27"} Nov 03 21:02:32.244271 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:32.244204 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf9b10d-ktw77" podStartSLOduration=2.244189087 podStartE2EDuration="2.244189087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:02:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:02:32.239149707 +0000 UTC m=+1387.464469498" watchObservedRunningTime="2026-11-03 21:02:32.244189087 +0000 UTC m=+1387.469508877" Nov 03 21:02:33.108844 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:33.108809 4305 generic.go:356] "Generic (PLEG): container finished" podID="2fa000fe-eafe-4adb-a4a7-28251831502d" containerID="b1f5fcca462a110a31f89fdad970f1c3c4cd83f7ed7e0e477fbabc4903226b93" exitCode=0 Nov 03 21:02:33.109295 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:33.108892 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf9b10d-ktw77" event={"ID":"2fa000fe-eafe-4adb-a4a7-28251831502d","Type":"ContainerDied","Data":"b1f5fcca462a110a31f89fdad970f1c3c4cd83f7ed7e0e477fbabc4903226b93"} Nov 03 21:02:34.299934 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:34.299897 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf9b10d-ktw77" Nov 03 21:02:34.313758 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:34.313724 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fd23f4f-5759r"] Nov 03 21:02:34.314526 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:34.314503 4305 cpu_manager.go:414] "containerMap: removing container" podUID="2fa000fe-eafe-4adb-a4a7-28251831502d" containerName="revoke-keys" Nov 03 21:02:34.314526 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:34.314526 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fa000fe-eafe-4adb-a4a7-28251831502d" containerName="revoke-keys" Nov 03 21:02:34.314739 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:34.314725 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fa000fe-eafe-4adb-a4a7-28251831502d" containerName="revoke-keys" Nov 03 21:02:34.318508 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:34.318485 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd23f4f-5759r" Nov 03 21:02:34.373655 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:34.373559 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fa000fe-eafe-4adb-a4a7-28251831502d-kube-api-access-x8h2p\" (UniqueName: \"kubernetes.io/projected/2fa000fe-eafe-4adb-a4a7-28251831502d-kube-api-access-x8h2p\") pod \"2fa000fe-eafe-4adb-a4a7-28251831502d\" (UID: \"2fa000fe-eafe-4adb-a4a7-28251831502d\") " Nov 03 21:02:34.373941 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:34.373916 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fa000fe-eafe-4adb-a4a7-28251831502d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fa000fe-eafe-4adb-a4a7-28251831502d-maas-api-xks-ca\") pod \"2fa000fe-eafe-4adb-a4a7-28251831502d\" (UID: \"2fa000fe-eafe-4adb-a4a7-28251831502d\") " Nov 03 21:02:34.374076 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:34.373967 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fa000fe-eafe-4adb-a4a7-28251831502d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fa000fe-eafe-4adb-a4a7-28251831502d-maas-api-openshift-ca\") pod \"2fa000fe-eafe-4adb-a4a7-28251831502d\" (UID: \"2fa000fe-eafe-4adb-a4a7-28251831502d\") " Nov 03 21:02:34.374475 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:34.374424 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fa000fe-eafe-4adb-a4a7-28251831502d-maas-api-openshift-ca" pod "2fa000fe-eafe-4adb-a4a7-28251831502d" (UID: "2fa000fe-eafe-4adb-a4a7-28251831502d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:02:34.375071 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:34.375043 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/878a0158-63b8-43da-8925-bb196f38da4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fd23f4f-5759r\" (UID: \"878a0158-63b8-43da-8925-bb196f38da4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd23f4f-5759r" Nov 03 21:02:34.375284 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:34.375255 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/878a0158-63b8-43da-8925-bb196f38da4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fd23f4f-5759r\" (UID: \"878a0158-63b8-43da-8925-bb196f38da4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd23f4f-5759r" Nov 03 21:02:34.375375 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:34.375317 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42hrn\" (UniqueName: \"kubernetes.io/projected/878a0158-63b8-43da-8925-bb196f38da4f-kube-api-access-42hrn\") pod \"maas-api-revoke-sub-1fd23f4f-5759r\" (UID: \"878a0158-63b8-43da-8925-bb196f38da4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd23f4f-5759r" Nov 03 21:02:34.375552 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:34.375537 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fa000fe-eafe-4adb-a4a7-28251831502d-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:02:34.376600 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:34.376539 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fa000fe-eafe-4adb-a4a7-28251831502d-kube-api-access-x8h2p" pod "2fa000fe-eafe-4adb-a4a7-28251831502d" (UID: "2fa000fe-eafe-4adb-a4a7-28251831502d"). InnerVolumeSpecName "kube-api-access-x8h2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:02:34.376741 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:34.376674 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fa000fe-eafe-4adb-a4a7-28251831502d-maas-api-xks-ca" pod "2fa000fe-eafe-4adb-a4a7-28251831502d" (UID: "2fa000fe-eafe-4adb-a4a7-28251831502d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:02:34.382762 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:34.382732 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fd23f4f-5759r"] Nov 03 21:02:34.477451 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:34.477403 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/878a0158-63b8-43da-8925-bb196f38da4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fd23f4f-5759r\" (UID: \"878a0158-63b8-43da-8925-bb196f38da4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd23f4f-5759r" Nov 03 21:02:34.477618 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:34.477584 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/878a0158-63b8-43da-8925-bb196f38da4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fd23f4f-5759r\" (UID: \"878a0158-63b8-43da-8925-bb196f38da4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd23f4f-5759r" Nov 03 21:02:34.477685 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:34.477617 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42hrn\" (UniqueName: \"kubernetes.io/projected/878a0158-63b8-43da-8925-bb196f38da4f-kube-api-access-42hrn\") pod \"maas-api-revoke-sub-1fd23f4f-5759r\" (UID: \"878a0158-63b8-43da-8925-bb196f38da4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd23f4f-5759r" Nov 03 21:02:34.477733 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:34.477690 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fa000fe-eafe-4adb-a4a7-28251831502d-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:02:34.477733 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:34.477702 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8h2p\" (UniqueName: \"kubernetes.io/projected/2fa000fe-eafe-4adb-a4a7-28251831502d-kube-api-access-x8h2p\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:02:34.478233 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:34.478210 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/878a0158-63b8-43da-8925-bb196f38da4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fd23f4f-5759r\" (UID: \"878a0158-63b8-43da-8925-bb196f38da4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd23f4f-5759r" Nov 03 21:02:34.479988 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:34.479971 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/878a0158-63b8-43da-8925-bb196f38da4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fd23f4f-5759r\" (UID: \"878a0158-63b8-43da-8925-bb196f38da4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd23f4f-5759r" Nov 03 21:02:34.543302 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:34.543271 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42hrn\" (UniqueName: \"kubernetes.io/projected/878a0158-63b8-43da-8925-bb196f38da4f-kube-api-access-42hrn\") pod \"maas-api-revoke-sub-1fd23f4f-5759r\" (UID: \"878a0158-63b8-43da-8925-bb196f38da4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd23f4f-5759r" Nov 03 21:02:34.633318 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:34.633235 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd23f4f-5759r" Nov 03 21:02:35.022246 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:35.022213 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fd23f4f-5759r"] Nov 03 21:02:35.022595 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:02:35.022568 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod878a0158_63b8_43da_8925_bb196f38da4f.slice/crio-6f719cb5531ee3878a1d44b362eb0c1cc1a536eb366918207e7b5964b9730dd4 WatchSource:0}: Error finding container 6f719cb5531ee3878a1d44b362eb0c1cc1a536eb366918207e7b5964b9730dd4: Status 404 returned error can't find the container with id 6f719cb5531ee3878a1d44b362eb0c1cc1a536eb366918207e7b5964b9730dd4 Nov 03 21:02:35.132765 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:35.132736 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf9b10d-ktw77" Nov 03 21:02:35.132938 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:35.132778 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf9b10d-ktw77" event={"ID":"2fa000fe-eafe-4adb-a4a7-28251831502d","Type":"ContainerDied","Data":"4ae7a7ed57e9bfd618269de5784d043355c9dc4ddb388cebb792431ccee1cd27"} Nov 03 21:02:35.132938 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:35.132836 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ae7a7ed57e9bfd618269de5784d043355c9dc4ddb388cebb792431ccee1cd27" Nov 03 21:02:35.134458 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:35.134419 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd23f4f-5759r" event={"ID":"878a0158-63b8-43da-8925-bb196f38da4f","Type":"ContainerStarted","Data":"6f719cb5531ee3878a1d44b362eb0c1cc1a536eb366918207e7b5964b9730dd4"} Nov 03 21:02:36.144334 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:36.144299 4305 generic.go:356] "Generic (PLEG): container finished" podID="878a0158-63b8-43da-8925-bb196f38da4f" containerID="4966a8b3aabf843074be11bd2088ce6affa5d3dac9edebad7bf98b2d1ccd0352" exitCode=0 Nov 03 21:02:36.144765 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:36.144384 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd23f4f-5759r" event={"ID":"878a0158-63b8-43da-8925-bb196f38da4f","Type":"ContainerDied","Data":"4966a8b3aabf843074be11bd2088ce6affa5d3dac9edebad7bf98b2d1ccd0352"} Nov 03 21:02:37.320936 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:37.320906 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd23f4f-5759r" Nov 03 21:02:37.424492 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:37.424361 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/878a0158-63b8-43da-8925-bb196f38da4f-kube-api-access-42hrn\" (UniqueName: \"kubernetes.io/projected/878a0158-63b8-43da-8925-bb196f38da4f-kube-api-access-42hrn\") pod \"878a0158-63b8-43da-8925-bb196f38da4f\" (UID: \"878a0158-63b8-43da-8925-bb196f38da4f\") " Nov 03 21:02:37.424492 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:37.424484 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/878a0158-63b8-43da-8925-bb196f38da4f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/878a0158-63b8-43da-8925-bb196f38da4f-maas-api-xks-ca\") pod \"878a0158-63b8-43da-8925-bb196f38da4f\" (UID: \"878a0158-63b8-43da-8925-bb196f38da4f\") " Nov 03 21:02:37.424746 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:37.424553 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/878a0158-63b8-43da-8925-bb196f38da4f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/878a0158-63b8-43da-8925-bb196f38da4f-maas-api-openshift-ca\") pod \"878a0158-63b8-43da-8925-bb196f38da4f\" (UID: \"878a0158-63b8-43da-8925-bb196f38da4f\") " Nov 03 21:02:37.424936 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:37.424904 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/878a0158-63b8-43da-8925-bb196f38da4f-maas-api-openshift-ca" pod "878a0158-63b8-43da-8925-bb196f38da4f" (UID: "878a0158-63b8-43da-8925-bb196f38da4f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:02:37.425210 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:37.425191 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/878a0158-63b8-43da-8925-bb196f38da4f-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:02:37.426840 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:37.426817 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/878a0158-63b8-43da-8925-bb196f38da4f-kube-api-access-42hrn" pod "878a0158-63b8-43da-8925-bb196f38da4f" (UID: "878a0158-63b8-43da-8925-bb196f38da4f"). InnerVolumeSpecName "kube-api-access-42hrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:02:37.426907 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:37.426820 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/878a0158-63b8-43da-8925-bb196f38da4f-maas-api-xks-ca" pod "878a0158-63b8-43da-8925-bb196f38da4f" (UID: "878a0158-63b8-43da-8925-bb196f38da4f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:02:37.527025 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:37.526982 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42hrn\" (UniqueName: \"kubernetes.io/projected/878a0158-63b8-43da-8925-bb196f38da4f-kube-api-access-42hrn\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:02:37.527025 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:37.527022 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/878a0158-63b8-43da-8925-bb196f38da4f-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:02:38.162207 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:38.162175 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd23f4f-5759r" Nov 03 21:02:38.162207 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:38.162198 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd23f4f-5759r" event={"ID":"878a0158-63b8-43da-8925-bb196f38da4f","Type":"ContainerDied","Data":"6f719cb5531ee3878a1d44b362eb0c1cc1a536eb366918207e7b5964b9730dd4"} Nov 03 21:02:38.162421 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:38.162259 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f719cb5531ee3878a1d44b362eb0c1cc1a536eb366918207e7b5964b9730dd4" Nov 03 21:02:40.477424 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:40.477369 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-46590df2-66f54d6b9f-b48rg"] Nov 03 21:02:40.478495 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:40.478468 4305 cpu_manager.go:414] "containerMap: removing container" podUID="878a0158-63b8-43da-8925-bb196f38da4f" containerName="revoke-keys" Nov 03 21:02:40.478495 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:40.478493 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="878a0158-63b8-43da-8925-bb196f38da4f" containerName="revoke-keys" Nov 03 21:02:40.478703 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:40.478683 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="878a0158-63b8-43da-8925-bb196f38da4f" containerName="revoke-keys" Nov 03 21:02:40.483133 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:40.483109 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-46590df2-66f54d6b9f-b48rg" Nov 03 21:02:40.499364 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:40.499330 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47183bab-92a7-49cc-a7ef-446106ac055d-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-46590df2-66f54d6b9f-b48rg\" (UID: \"47183bab-92a7-49cc-a7ef-446106ac055d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-46590df2-66f54d6b9f-b48rg" Nov 03 21:02:40.499688 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:40.499667 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-495xj\" (UniqueName: \"kubernetes.io/projected/47183bab-92a7-49cc-a7ef-446106ac055d-kube-api-access-495xj\") pod \"payload-pre-processing-e2e-migrate-46590df2-66f54d6b9f-b48rg\" (UID: \"47183bab-92a7-49cc-a7ef-446106ac055d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-46590df2-66f54d6b9f-b48rg" Nov 03 21:02:40.538686 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:40.538654 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-46590df2-66f54d6b9f-b48rg"] Nov 03 21:02:40.607649 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:40.607617 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-46590df2\"" Nov 03 21:02:40.608570 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:40.608094 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-46590df2-dockercfg-9jr7n\"" Nov 03 21:02:40.610072 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:40.610045 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-495xj\" (UniqueName: \"kubernetes.io/projected/47183bab-92a7-49cc-a7ef-446106ac055d-kube-api-access-495xj\") pod \"payload-pre-processing-e2e-migrate-46590df2-66f54d6b9f-b48rg\" (UID: \"47183bab-92a7-49cc-a7ef-446106ac055d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-46590df2-66f54d6b9f-b48rg" Nov 03 21:02:40.639709 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:40.639673 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47183bab-92a7-49cc-a7ef-446106ac055d-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-46590df2-66f54d6b9f-b48rg\" (UID: \"47183bab-92a7-49cc-a7ef-446106ac055d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-46590df2-66f54d6b9f-b48rg" Nov 03 21:02:40.643721 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:40.643666 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47183bab-92a7-49cc-a7ef-446106ac055d-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-46590df2-66f54d6b9f-b48rg\" (UID: \"47183bab-92a7-49cc-a7ef-446106ac055d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-46590df2-66f54d6b9f-b48rg" Nov 03 21:02:40.653887 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:40.653529 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-46590df2-7d46fd668-6cc2h"] Nov 03 21:02:40.709578 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:40.709414 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-46590df2-7d46fd668-6cc2h"] Nov 03 21:02:40.709803 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:40.709630 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-46590df2-7d46fd668-6cc2h" Nov 03 21:02:40.876499 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:40.874578 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-495xj\" (UniqueName: \"kubernetes.io/projected/47183bab-92a7-49cc-a7ef-446106ac055d-kube-api-access-495xj\") pod \"payload-pre-processing-e2e-migrate-46590df2-66f54d6b9f-b48rg\" (UID: \"47183bab-92a7-49cc-a7ef-446106ac055d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-46590df2-66f54d6b9f-b48rg" Nov 03 21:02:40.886965 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:40.886894 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3320ca8a-f6ac-44c1-a877-a62db11a904c-plugins-config-volume\") pod \"payload-processing-e2e-migrate-46590df2-7d46fd668-6cc2h\" (UID: \"3320ca8a-f6ac-44c1-a877-a62db11a904c\") " pod="openshift-ingress/payload-processing-e2e-migrate-46590df2-7d46fd668-6cc2h" Nov 03 21:02:40.887137 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:40.887014 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mqf7\" (UniqueName: \"kubernetes.io/projected/3320ca8a-f6ac-44c1-a877-a62db11a904c-kube-api-access-9mqf7\") pod \"payload-processing-e2e-migrate-46590df2-7d46fd668-6cc2h\" (UID: \"3320ca8a-f6ac-44c1-a877-a62db11a904c\") " pod="openshift-ingress/payload-processing-e2e-migrate-46590df2-7d46fd668-6cc2h" Nov 03 21:02:40.989047 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:40.989005 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3320ca8a-f6ac-44c1-a877-a62db11a904c-plugins-config-volume\") pod \"payload-processing-e2e-migrate-46590df2-7d46fd668-6cc2h\" (UID: \"3320ca8a-f6ac-44c1-a877-a62db11a904c\") " pod="openshift-ingress/payload-processing-e2e-migrate-46590df2-7d46fd668-6cc2h" Nov 03 21:02:40.989269 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:40.989086 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mqf7\" (UniqueName: \"kubernetes.io/projected/3320ca8a-f6ac-44c1-a877-a62db11a904c-kube-api-access-9mqf7\") pod \"payload-processing-e2e-migrate-46590df2-7d46fd668-6cc2h\" (UID: \"3320ca8a-f6ac-44c1-a877-a62db11a904c\") " pod="openshift-ingress/payload-processing-e2e-migrate-46590df2-7d46fd668-6cc2h" Nov 03 21:02:40.989872 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:40.989850 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3320ca8a-f6ac-44c1-a877-a62db11a904c-plugins-config-volume\") pod \"payload-processing-e2e-migrate-46590df2-7d46fd668-6cc2h\" (UID: \"3320ca8a-f6ac-44c1-a877-a62db11a904c\") " pod="openshift-ingress/payload-processing-e2e-migrate-46590df2-7d46fd668-6cc2h" Nov 03 21:02:41.050993 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:41.050965 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mqf7\" (UniqueName: \"kubernetes.io/projected/3320ca8a-f6ac-44c1-a877-a62db11a904c-kube-api-access-9mqf7\") pod \"payload-processing-e2e-migrate-46590df2-7d46fd668-6cc2h\" (UID: \"3320ca8a-f6ac-44c1-a877-a62db11a904c\") " pod="openshift-ingress/payload-processing-e2e-migrate-46590df2-7d46fd668-6cc2h" Nov 03 21:02:41.102348 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:41.102254 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-46590df2-66f54d6b9f-b48rg" Nov 03 21:02:41.147316 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:41.146751 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-46590df2-7d46fd668-6cc2h" Nov 03 21:02:41.512125 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:41.512095 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2df98d9-8zlwf"] Nov 03 21:02:41.517993 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:41.517970 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2df98d9-8zlwf" Nov 03 21:02:41.561411 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:02:41.561374 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47183bab_92a7_49cc_a7ef_446106ac055d.slice/crio-3c8fc1452504380e2834582260ee2e00cd49af7acc0f4df5f7b20c8a3fd030f4 WatchSource:0}: Error finding container 3c8fc1452504380e2834582260ee2e00cd49af7acc0f4df5f7b20c8a3fd030f4: Status 404 returned error can't find the container with id 3c8fc1452504380e2834582260ee2e00cd49af7acc0f4df5f7b20c8a3fd030f4 Nov 03 21:02:41.561597 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:41.561493 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-46590df2-66f54d6b9f-b48rg"] Nov 03 21:02:41.572352 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:41.572328 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:02:41.572514 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:41.572404 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:02:41.572590 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:41.572548 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:02:41.573535 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:41.572861 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77vtd\"" Nov 03 21:02:41.575251 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:41.575229 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2df98d9-8zlwf"] Nov 03 21:02:41.610304 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:41.610274 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-46590df2-7d46fd668-6cc2h"] Nov 03 21:02:41.614899 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:41.614874 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cff10ef-287d-40dd-a147-2b928fc7d053-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2df98d9-8zlwf\" (UID: \"2cff10ef-287d-40dd-a147-2b928fc7d053\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2df98d9-8zlwf" Nov 03 21:02:41.615180 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:41.615156 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpfwn\" (UniqueName: \"kubernetes.io/projected/2cff10ef-287d-40dd-a147-2b928fc7d053-kube-api-access-zpfwn\") pod \"maas-api-revoke-sub-f2df98d9-8zlwf\" (UID: \"2cff10ef-287d-40dd-a147-2b928fc7d053\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2df98d9-8zlwf" Nov 03 21:02:41.615381 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:41.615363 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cff10ef-287d-40dd-a147-2b928fc7d053-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2df98d9-8zlwf\" (UID: \"2cff10ef-287d-40dd-a147-2b928fc7d053\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2df98d9-8zlwf" Nov 03 21:02:41.624261 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:02:41.624232 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3320ca8a_f6ac_44c1_a877_a62db11a904c.slice/crio-41f3e949d5314731d4cbfe47029bf1a30db3404f772f6b6cbafc725dccfe4690 WatchSource:0}: Error finding container 41f3e949d5314731d4cbfe47029bf1a30db3404f772f6b6cbafc725dccfe4690: Status 404 returned error can't find the container with id 41f3e949d5314731d4cbfe47029bf1a30db3404f772f6b6cbafc725dccfe4690 Nov 03 21:02:41.716469 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:41.716413 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cff10ef-287d-40dd-a147-2b928fc7d053-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2df98d9-8zlwf\" (UID: \"2cff10ef-287d-40dd-a147-2b928fc7d053\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2df98d9-8zlwf" Nov 03 21:02:41.716641 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:41.716512 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cff10ef-287d-40dd-a147-2b928fc7d053-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2df98d9-8zlwf\" (UID: \"2cff10ef-287d-40dd-a147-2b928fc7d053\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2df98d9-8zlwf" Nov 03 21:02:41.716780 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:41.716761 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpfwn\" (UniqueName: \"kubernetes.io/projected/2cff10ef-287d-40dd-a147-2b928fc7d053-kube-api-access-zpfwn\") pod \"maas-api-revoke-sub-f2df98d9-8zlwf\" (UID: \"2cff10ef-287d-40dd-a147-2b928fc7d053\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2df98d9-8zlwf" Nov 03 21:02:41.717212 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:41.717122 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cff10ef-287d-40dd-a147-2b928fc7d053-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2df98d9-8zlwf\" (UID: \"2cff10ef-287d-40dd-a147-2b928fc7d053\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2df98d9-8zlwf" Nov 03 21:02:41.718977 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:41.718954 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cff10ef-287d-40dd-a147-2b928fc7d053-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2df98d9-8zlwf\" (UID: \"2cff10ef-287d-40dd-a147-2b928fc7d053\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2df98d9-8zlwf" Nov 03 21:02:41.778325 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:41.778241 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpfwn\" (UniqueName: \"kubernetes.io/projected/2cff10ef-287d-40dd-a147-2b928fc7d053-kube-api-access-zpfwn\") pod \"maas-api-revoke-sub-f2df98d9-8zlwf\" (UID: \"2cff10ef-287d-40dd-a147-2b928fc7d053\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2df98d9-8zlwf" Nov 03 21:02:41.833891 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:41.833844 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2df98d9-8zlwf" Nov 03 21:02:42.225754 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:42.225714 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-46590df2-7d46fd668-6cc2h" event={"ID":"3320ca8a-f6ac-44c1-a877-a62db11a904c","Type":"ContainerStarted","Data":"9aa6207e126cab6ebd5198196fee86190fa28c39be32f9335bf3fcbd4246c5b8"} Nov 03 21:02:42.226017 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:42.225791 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-46590df2-7d46fd668-6cc2h" Nov 03 21:02:42.226017 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:42.225814 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-46590df2-7d46fd668-6cc2h" event={"ID":"3320ca8a-f6ac-44c1-a877-a62db11a904c","Type":"ContainerStarted","Data":"41f3e949d5314731d4cbfe47029bf1a30db3404f772f6b6cbafc725dccfe4690"} Nov 03 21:02:42.229014 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:42.228977 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-46590df2-66f54d6b9f-b48rg" event={"ID":"47183bab-92a7-49cc-a7ef-446106ac055d","Type":"ContainerStarted","Data":"5c51283e79e4efe9e1f01bd9a15a700d3d4dace319264c013304ffba19666ca9"} Nov 03 21:02:42.229188 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:42.229043 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-46590df2-66f54d6b9f-b48rg" event={"ID":"47183bab-92a7-49cc-a7ef-446106ac055d","Type":"ContainerStarted","Data":"3c8fc1452504380e2834582260ee2e00cd49af7acc0f4df5f7b20c8a3fd030f4"} Nov 03 21:02:42.229698 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:42.229679 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-46590df2-66f54d6b9f-b48rg" Nov 03 21:02:42.236246 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:42.236217 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2df98d9-8zlwf"] Nov 03 21:02:42.240864 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:02:42.240835 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cff10ef_287d_40dd_a147_2b928fc7d053.slice/crio-346020505c33c13b8dbd19d65e459e9671ec92afbf694f7f9fe9f488e5d65395 WatchSource:0}: Error finding container 346020505c33c13b8dbd19d65e459e9671ec92afbf694f7f9fe9f488e5d65395: Status 404 returned error can't find the container with id 346020505c33c13b8dbd19d65e459e9671ec92afbf694f7f9fe9f488e5d65395 Nov 03 21:02:42.346412 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:42.346356 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-46590df2-7d46fd668-6cc2h" podStartSLOduration=2.346341611 podStartE2EDuration="2.346341611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:02:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:02:42.342108823 +0000 UTC m=+1397.567428617" watchObservedRunningTime="2026-11-03 21:02:42.346341611 +0000 UTC m=+1397.571661401" Nov 03 21:02:42.516676 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:42.515621 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-46590df2-66f54d6b9f-b48rg" podStartSLOduration=2.515598335 podStartE2EDuration="2.515598335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:02:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:02:42.464876342 +0000 UTC m=+1397.690196134" watchObservedRunningTime="2026-11-03 21:02:42.515598335 +0000 UTC m=+1397.740918128" Nov 03 21:02:43.277534 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:43.277494 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2df98d9-8zlwf" event={"ID":"2cff10ef-287d-40dd-a147-2b928fc7d053","Type":"ContainerStarted","Data":"38d7749aaa68308fd0a5db54d28456f32902deb2f2ffff6199326705de114ca8"} Nov 03 21:02:43.277878 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:43.277851 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2df98d9-8zlwf" event={"ID":"2cff10ef-287d-40dd-a147-2b928fc7d053","Type":"ContainerStarted","Data":"346020505c33c13b8dbd19d65e459e9671ec92afbf694f7f9fe9f488e5d65395"} Nov 03 21:02:43.412226 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:43.412108 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2df98d9-8zlwf" podStartSLOduration=2.412086876 podStartE2EDuration="2.412086876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:02:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:02:43.400809747 +0000 UTC m=+1398.626129539" watchObservedRunningTime="2026-11-03 21:02:43.412086876 +0000 UTC m=+1398.637406665" Nov 03 21:02:44.085524 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:02:44.074078 4305 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cff10ef_287d_40dd_a147_2b928fc7d053.slice/crio-conmon-38d7749aaa68308fd0a5db54d28456f32902deb2f2ffff6199326705de114ca8.scope\": RecentStats: unable to find data in memory cache]" Nov 03 21:02:44.307458 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:44.307340 4305 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" podUID="7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:44.411470 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:44.409957 4305 generic.go:356] "Generic (PLEG): container finished" podID="2cff10ef-287d-40dd-a147-2b928fc7d053" containerID="38d7749aaa68308fd0a5db54d28456f32902deb2f2ffff6199326705de114ca8" exitCode=0 Nov 03 21:02:44.411705 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:44.411500 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2df98d9-8zlwf" event={"ID":"2cff10ef-287d-40dd-a147-2b928fc7d053","Type":"ContainerDied","Data":"38d7749aaa68308fd0a5db54d28456f32902deb2f2ffff6199326705de114ca8"} Nov 03 21:02:44.720628 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:44.720530 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:02:44.720912 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:44.720880 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s" podUID="c7f2f4c8-63bb-4a07-8860-91a420ebc1ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:44.962457 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:44.947527 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" podUID="be06bc96-808a-46e6-9872-01baf1a11a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:44.987517 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:44.986666 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" podUID="8ee1f028-8503-4034-9954-eb06fcf402e7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:45.034591 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:45.034544 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" podUID="7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:45.035233 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:45.035200 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" podUID="8793f095-97b1-4e0a-8382-b541f90a0de2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 03 21:02:45.175601 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:45.175538 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" podUID="63a704b4-51ce-46c3-8007-34b308aff332" containerName="main" probeResult="failure" output="Get \"https://10.134.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:45.176502 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:45.176240 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" podUID="d5d3dcaa-a51a-40c6-8049-871b300552e1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:45.228448 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:45.228387 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" podUID="7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:45.355522 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:45.354638 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" podUID="006c5eae-06af-447b-a1d8-cd719cce2a92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:45.483179 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:45.483146 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d830196-vhlrr"] Nov 03 21:02:45.561464 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:45.508543 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" podUID="793b8ca3-3a2e-49a1-9be4-4971ba59df09" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:45.650553 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:45.562773 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" podUID="6e7f30a9-0862-48a5-9ab7-e692e75bb294" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 03 21:02:45.650553 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:45.567829 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" podUID="80de7944-70f6-49e1-a0e8-4b82d5caad0d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Nov 03 21:02:45.650553 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:45.592853 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d830196-vhlrr" Nov 03 21:02:45.955523 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:45.923896 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7cpf\" (UniqueName: \"kubernetes.io/projected/948e2234-c69a-47c6-b4f1-6d61d8ac19dd-kube-api-access-v7cpf\") pod \"maas-api-revoke-sub-7d830196-vhlrr\" (UID: \"948e2234-c69a-47c6-b4f1-6d61d8ac19dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d830196-vhlrr" Nov 03 21:02:45.997499 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:45.977579 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" podUID="be06bc96-808a-46e6-9872-01baf1a11a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:46.027215 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:45.988702 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" podUID="8ee1f028-8503-4034-9954-eb06fcf402e7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:46.028676 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:45.997862 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/948e2234-c69a-47c6-b4f1-6d61d8ac19dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d830196-vhlrr\" (UID: \"948e2234-c69a-47c6-b4f1-6d61d8ac19dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d830196-vhlrr" Nov 03 21:02:46.028907 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:46.028887 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/948e2234-c69a-47c6-b4f1-6d61d8ac19dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d830196-vhlrr\" (UID: \"948e2234-c69a-47c6-b4f1-6d61d8ac19dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d830196-vhlrr" Nov 03 21:02:46.056649 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:46.056592 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" podUID="7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 03 21:02:46.056883 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:46.056645 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" podUID="8793f095-97b1-4e0a-8382-b541f90a0de2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:46.082713 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:46.058149 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" Nov 03 21:02:46.082713 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:46.058217 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" Nov 03 21:02:46.082713 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:46.058239 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d830196-vhlrr"] Nov 03 21:02:46.082713 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:46.058304 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" Nov 03 21:02:46.082713 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:46.058366 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" Nov 03 21:02:46.132098 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:46.132025 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/948e2234-c69a-47c6-b4f1-6d61d8ac19dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d830196-vhlrr\" (UID: \"948e2234-c69a-47c6-b4f1-6d61d8ac19dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d830196-vhlrr" Nov 03 21:02:46.132522 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:46.132478 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/948e2234-c69a-47c6-b4f1-6d61d8ac19dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d830196-vhlrr\" (UID: \"948e2234-c69a-47c6-b4f1-6d61d8ac19dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d830196-vhlrr" Nov 03 21:02:46.151527 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:46.133276 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7cpf\" (UniqueName: \"kubernetes.io/projected/948e2234-c69a-47c6-b4f1-6d61d8ac19dd-kube-api-access-v7cpf\") pod \"maas-api-revoke-sub-7d830196-vhlrr\" (UID: \"948e2234-c69a-47c6-b4f1-6d61d8ac19dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d830196-vhlrr" Nov 03 21:02:46.151928 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:46.136856 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/948e2234-c69a-47c6-b4f1-6d61d8ac19dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d830196-vhlrr\" (UID: \"948e2234-c69a-47c6-b4f1-6d61d8ac19dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d830196-vhlrr" Nov 03 21:02:46.199466 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:46.195958 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" podUID="d5d3dcaa-a51a-40c6-8049-871b300552e1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:46.212105 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:46.200556 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" podUID="63a704b4-51ce-46c3-8007-34b308aff332" containerName="main" probeResult="failure" output="Get \"https://10.134.0.101:8000/ready\": context deadline exceeded" Nov 03 21:02:46.226656 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:46.212427 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" Nov 03 21:02:46.227043 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:46.227009 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" Nov 03 21:02:46.243664 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:46.240549 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" podUID="7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 03 21:02:46.277510 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:46.250631 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" Nov 03 21:02:46.277510 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:46.241735 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:02:46.277510 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:46.251703 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d" podUID="d9103e89-52f2-4022-bd9b-dc554c82f479" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:46.421463 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:46.355485 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7cpf\" (UniqueName: \"kubernetes.io/projected/948e2234-c69a-47c6-b4f1-6d61d8ac19dd-kube-api-access-v7cpf\") pod \"maas-api-revoke-sub-7d830196-vhlrr\" (UID: \"948e2234-c69a-47c6-b4f1-6d61d8ac19dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d830196-vhlrr" Nov 03 21:02:46.421846 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:46.421799 4305 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" podUID="8ee1f028-8503-4034-9954-eb06fcf402e7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/health\": context deadline exceeded" Nov 03 21:02:46.532524 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:46.517852 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" podUID="006c5eae-06af-447b-a1d8-cd719cce2a92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:46.533523 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:46.533477 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" podUID="793b8ca3-3a2e-49a1-9be4-4971ba59df09" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 03 21:02:46.611740 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:46.546733 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" Nov 03 21:02:46.611740 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:46.546805 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" Nov 03 21:02:46.611740 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:46.609617 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:02:46.611740 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:46.609721 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd" podUID="94bcfeb9-714d-4f25-8ce7-c6c4577ea837" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:46.612268 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:46.612230 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" podUID="6e7f30a9-0862-48a5-9ab7-e692e75bb294" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 03 21:02:46.612399 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:46.612188 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" podUID="80de7944-70f6-49e1-a0e8-4b82d5caad0d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:46.634773 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:46.613010 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" Nov 03 21:02:46.634773 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:46.622255 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" Nov 03 21:02:47.053193 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:47.053153 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" podUID="8ee1f028-8503-4034-9954-eb06fcf402e7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:47.057930 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:47.054733 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/948e2234-c69a-47c6-b4f1-6d61d8ac19dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d830196-vhlrr\" (UID: \"948e2234-c69a-47c6-b4f1-6d61d8ac19dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d830196-vhlrr" Nov 03 21:02:47.081223 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:47.080233 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" podUID="8793f095-97b1-4e0a-8382-b541f90a0de2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 03 21:02:47.081824 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:47.081095 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" podUID="7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:47.141667 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:47.101286 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" podUID="be06bc96-808a-46e6-9872-01baf1a11a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.107:8000/ready\": context deadline exceeded" Nov 03 21:02:47.142486 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:47.118948 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d830196-vhlrr" Nov 03 21:02:47.272533 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:47.202396 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" podUID="d5d3dcaa-a51a-40c6-8049-871b300552e1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:47.493707 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:47.204526 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" podUID="63a704b4-51ce-46c3-8007-34b308aff332" containerName="main" probeResult="failure" output="Get \"https://10.134.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:47.498751 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:47.301540 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" podUID="7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:47.499463 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:47.364543 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:02:47.499693 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:47.499662 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28" podUID="58829723-0628-457e-80b8-f81d15ecfe2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:47.499858 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:47.470981 4305 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" podUID="8793f095-97b1-4e0a-8382-b541f90a0de2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": context deadline exceeded" Nov 03 21:02:47.499991 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:47.489987 4305 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" podUID="80de7944-70f6-49e1-a0e8-4b82d5caad0d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": context deadline exceeded" Nov 03 21:02:47.500127 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:02:47.416594 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:02:47.500240 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:47.499403 4305 patch_prober.go:29] interesting pod/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:02:47.500362 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:47.500343 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p" podUID="e4b14774-55f2-4fad-88ad-74d08bf328f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:47.611580 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:47.611525 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" podUID="793b8ca3-3a2e-49a1-9be4-4971ba59df09" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:47.659294 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:47.659246 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" podUID="006c5eae-06af-447b-a1d8-cd719cce2a92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Nov 03 21:02:47.718014 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:47.717963 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" podUID="80de7944-70f6-49e1-a0e8-4b82d5caad0d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:47.719338 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:47.718464 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" podUID="6e7f30a9-0862-48a5-9ab7-e692e75bb294" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:47.978642 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:47.978577 4305 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" podUID="63a704b4-51ce-46c3-8007-34b308aff332" containerName="main" probeResult="failure" output="Get \"https://10.134.0.101:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:48.250470 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:48.237072 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" podUID="8ee1f028-8503-4034-9954-eb06fcf402e7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": context deadline exceeded" Nov 03 21:02:48.291471 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:48.276544 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" podUID="be06bc96-808a-46e6-9872-01baf1a11a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:48.365467 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:48.365304 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" podUID="8793f095-97b1-4e0a-8382-b541f90a0de2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 03 21:02:48.390480 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:48.372946 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" podUID="7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 03 21:02:48.517585 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:48.491575 4305 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pblg" podUID="997eb6ae-52a1-4600-b955-59bdea0a6da3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:48.841637 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:48.827969 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" podUID="6e7f30a9-0862-48a5-9ab7-e692e75bb294" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 03 21:02:49.030561 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:49.030521 4305 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" podUID="793b8ca3-3a2e-49a1-9be4-4971ba59df09" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:49.127725 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:49.127406 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" podUID="7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:49.128240 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:49.128210 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" podUID="d5d3dcaa-a51a-40c6-8049-871b300552e1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded" Nov 03 21:02:49.144616 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:49.140610 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" podUID="006c5eae-06af-447b-a1d8-cd719cce2a92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Nov 03 21:02:49.144616 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:49.143007 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" podUID="80de7944-70f6-49e1-a0e8-4b82d5caad0d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Nov 03 21:02:49.188810 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:49.188749 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" podUID="63a704b4-51ce-46c3-8007-34b308aff332" containerName="main" probeResult="failure" output="Get \"https://10.134.0.101:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:49.199200 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:49.199156 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" podUID="793b8ca3-3a2e-49a1-9be4-4971ba59df09" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 03 21:02:49.287498 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:49.284802 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" podUID="be06bc96-808a-46e6-9872-01baf1a11a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:49.298472 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:49.291897 4305 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" podUID="6e7f30a9-0862-48a5-9ab7-e692e75bb294" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:49.299962 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:49.292645 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" podUID="8ee1f028-8503-4034-9954-eb06fcf402e7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": context deadline exceeded" Nov 03 21:02:49.375307 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:49.366228 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" podUID="8793f095-97b1-4e0a-8382-b541f90a0de2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:49.375307 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:49.374711 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" podUID="7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 03 21:02:49.664738 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:49.659731 4305 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" podUID="be06bc96-808a-46e6-9872-01baf1a11a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.107:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:02:49.731984 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:02:49.729030 4305 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.709s" Nov 03 21:02:49.731984 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:49.729110 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" Nov 03 21:02:49.775781 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:49.768940 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" Nov 03 21:02:49.857341 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:49.808722 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2df98d9-8zlwf" Nov 03 21:02:49.878054 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:49.878025 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" Nov 03 21:02:49.878326 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:49.878311 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" Nov 03 21:02:49.878501 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:49.878490 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" Nov 03 21:02:49.878659 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:49.878640 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2df98d9-8zlwf" event={"ID":"2cff10ef-287d-40dd-a147-2b928fc7d053","Type":"ContainerDied","Data":"346020505c33c13b8dbd19d65e459e9671ec92afbf694f7f9fe9f488e5d65395"} Nov 03 21:02:49.878774 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:49.878762 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="346020505c33c13b8dbd19d65e459e9671ec92afbf694f7f9fe9f488e5d65395" Nov 03 21:02:49.878904 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:49.878893 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" Nov 03 21:02:49.879061 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:49.879050 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" Nov 03 21:02:49.879165 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:49.879155 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" Nov 03 21:02:49.879266 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:49.879258 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" Nov 03 21:02:49.879373 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:49.879364 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" Nov 03 21:02:49.879505 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:49.879494 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" Nov 03 21:02:49.969241 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:49.969141 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cff10ef-287d-40dd-a147-2b928fc7d053-kube-api-access-zpfwn\" (UniqueName: \"kubernetes.io/projected/2cff10ef-287d-40dd-a147-2b928fc7d053-kube-api-access-zpfwn\") pod \"2cff10ef-287d-40dd-a147-2b928fc7d053\" (UID: \"2cff10ef-287d-40dd-a147-2b928fc7d053\") " Nov 03 21:02:49.969415 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:49.969252 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cff10ef-287d-40dd-a147-2b928fc7d053-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cff10ef-287d-40dd-a147-2b928fc7d053-maas-api-xks-ca\") pod \"2cff10ef-287d-40dd-a147-2b928fc7d053\" (UID: \"2cff10ef-287d-40dd-a147-2b928fc7d053\") " Nov 03 21:02:49.969415 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:49.969326 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cff10ef-287d-40dd-a147-2b928fc7d053-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cff10ef-287d-40dd-a147-2b928fc7d053-maas-api-openshift-ca\") pod \"2cff10ef-287d-40dd-a147-2b928fc7d053\" (UID: \"2cff10ef-287d-40dd-a147-2b928fc7d053\") " Nov 03 21:02:49.973331 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:49.973293 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cff10ef-287d-40dd-a147-2b928fc7d053-maas-api-openshift-ca" pod "2cff10ef-287d-40dd-a147-2b928fc7d053" (UID: "2cff10ef-287d-40dd-a147-2b928fc7d053"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:02:49.985699 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:49.985658 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cff10ef-287d-40dd-a147-2b928fc7d053-maas-api-xks-ca" pod "2cff10ef-287d-40dd-a147-2b928fc7d053" (UID: "2cff10ef-287d-40dd-a147-2b928fc7d053"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:02:49.986645 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:49.986610 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cff10ef-287d-40dd-a147-2b928fc7d053-kube-api-access-zpfwn" pod "2cff10ef-287d-40dd-a147-2b928fc7d053" (UID: "2cff10ef-287d-40dd-a147-2b928fc7d053"). InnerVolumeSpecName "kube-api-access-zpfwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:02:50.073147 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:50.073111 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpfwn\" (UniqueName: \"kubernetes.io/projected/2cff10ef-287d-40dd-a147-2b928fc7d053-kube-api-access-zpfwn\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:02:50.073147 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:50.073147 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cff10ef-287d-40dd-a147-2b928fc7d053-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:02:50.073363 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:50.073160 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cff10ef-287d-40dd-a147-2b928fc7d053-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:02:50.167459 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:50.157087 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d830196-vhlrr"] Nov 03 21:02:50.889113 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:50.889077 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2df98d9-8zlwf" Nov 03 21:02:50.897613 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:50.897588 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d830196-vhlrr" event={"ID":"948e2234-c69a-47c6-b4f1-6d61d8ac19dd","Type":"ContainerStarted","Data":"d02f9ebaf3fb8ca133e33c3a2ca5a2c7933aea686a87a2ec2e2eae119da51e57"} Nov 03 21:02:50.897713 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:50.897629 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d830196-vhlrr" event={"ID":"948e2234-c69a-47c6-b4f1-6d61d8ac19dd","Type":"ContainerStarted","Data":"78c7b43afd6374507696fcfd93f0f58ead5329473fe19fbf5786c9059a810a8e"} Nov 03 21:02:51.558948 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:51.558842 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d830196-vhlrr" podStartSLOduration=6.558826684 podStartE2EDuration="6.558826684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:02:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:02:51.555082993 +0000 UTC m=+1406.780402819" watchObservedRunningTime="2026-11-03 21:02:51.558826684 +0000 UTC m=+1406.784146474" Nov 03 21:02:51.900601 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:51.900566 4305 generic.go:356] "Generic (PLEG): container finished" podID="948e2234-c69a-47c6-b4f1-6d61d8ac19dd" containerID="d02f9ebaf3fb8ca133e33c3a2ca5a2c7933aea686a87a2ec2e2eae119da51e57" exitCode=0 Nov 03 21:02:51.901038 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:51.900606 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d830196-vhlrr" event={"ID":"948e2234-c69a-47c6-b4f1-6d61d8ac19dd","Type":"ContainerDied","Data":"d02f9ebaf3fb8ca133e33c3a2ca5a2c7933aea686a87a2ec2e2eae119da51e57"} Nov 03 21:02:53.112912 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:53.112881 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d830196-vhlrr" Nov 03 21:02:53.126410 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:53.126379 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/948e2234-c69a-47c6-b4f1-6d61d8ac19dd-kube-api-access-v7cpf\" (UniqueName: \"kubernetes.io/projected/948e2234-c69a-47c6-b4f1-6d61d8ac19dd-kube-api-access-v7cpf\") pod \"948e2234-c69a-47c6-b4f1-6d61d8ac19dd\" (UID: \"948e2234-c69a-47c6-b4f1-6d61d8ac19dd\") " Nov 03 21:02:53.126618 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:53.126420 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/948e2234-c69a-47c6-b4f1-6d61d8ac19dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/948e2234-c69a-47c6-b4f1-6d61d8ac19dd-maas-api-openshift-ca\") pod \"948e2234-c69a-47c6-b4f1-6d61d8ac19dd\" (UID: \"948e2234-c69a-47c6-b4f1-6d61d8ac19dd\") " Nov 03 21:02:53.126688 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:53.126655 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/948e2234-c69a-47c6-b4f1-6d61d8ac19dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/948e2234-c69a-47c6-b4f1-6d61d8ac19dd-maas-api-xks-ca\") pod \"948e2234-c69a-47c6-b4f1-6d61d8ac19dd\" (UID: \"948e2234-c69a-47c6-b4f1-6d61d8ac19dd\") " Nov 03 21:02:53.126747 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:53.126725 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/948e2234-c69a-47c6-b4f1-6d61d8ac19dd-maas-api-openshift-ca" pod "948e2234-c69a-47c6-b4f1-6d61d8ac19dd" (UID: "948e2234-c69a-47c6-b4f1-6d61d8ac19dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:02:53.127487 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:53.127462 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/948e2234-c69a-47c6-b4f1-6d61d8ac19dd-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:02:53.131830 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:53.131804 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4ad2ebc-hxffw"] Nov 03 21:02:53.131979 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:53.131876 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/948e2234-c69a-47c6-b4f1-6d61d8ac19dd-kube-api-access-v7cpf" pod "948e2234-c69a-47c6-b4f1-6d61d8ac19dd" (UID: "948e2234-c69a-47c6-b4f1-6d61d8ac19dd"). InnerVolumeSpecName "kube-api-access-v7cpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:02:53.131979 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:53.131950 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/948e2234-c69a-47c6-b4f1-6d61d8ac19dd-maas-api-xks-ca" pod "948e2234-c69a-47c6-b4f1-6d61d8ac19dd" (UID: "948e2234-c69a-47c6-b4f1-6d61d8ac19dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:02:53.132633 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:53.132614 4305 cpu_manager.go:414] "containerMap: removing container" podUID="948e2234-c69a-47c6-b4f1-6d61d8ac19dd" containerName="revoke-keys" Nov 03 21:02:53.132633 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:53.132631 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="948e2234-c69a-47c6-b4f1-6d61d8ac19dd" containerName="revoke-keys" Nov 03 21:02:53.132781 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:53.132648 4305 cpu_manager.go:414] "containerMap: removing container" podUID="2cff10ef-287d-40dd-a147-2b928fc7d053" containerName="revoke-keys" Nov 03 21:02:53.132781 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:53.132653 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cff10ef-287d-40dd-a147-2b928fc7d053" containerName="revoke-keys" Nov 03 21:02:53.132781 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:53.132749 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="948e2234-c69a-47c6-b4f1-6d61d8ac19dd" containerName="revoke-keys" Nov 03 21:02:53.132781 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:53.132761 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cff10ef-287d-40dd-a147-2b928fc7d053" containerName="revoke-keys" Nov 03 21:02:53.136881 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:53.136864 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ad2ebc-hxffw" Nov 03 21:02:53.190799 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:53.190767 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4ad2ebc-hxffw"] Nov 03 21:02:53.228645 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:53.228605 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5mgk\" (UniqueName: \"kubernetes.io/projected/6b7fadf9-8365-466e-925f-c4fb8ff49955-kube-api-access-x5mgk\") pod \"maas-api-revoke-sub-d4ad2ebc-hxffw\" (UID: \"6b7fadf9-8365-466e-925f-c4fb8ff49955\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ad2ebc-hxffw" Nov 03 21:02:53.228830 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:53.228653 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b7fadf9-8365-466e-925f-c4fb8ff49955-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4ad2ebc-hxffw\" (UID: \"6b7fadf9-8365-466e-925f-c4fb8ff49955\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ad2ebc-hxffw" Nov 03 21:02:53.228993 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:53.228962 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b7fadf9-8365-466e-925f-c4fb8ff49955-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4ad2ebc-hxffw\" (UID: \"6b7fadf9-8365-466e-925f-c4fb8ff49955\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ad2ebc-hxffw" Nov 03 21:02:53.229293 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:53.229272 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7cpf\" (UniqueName: \"kubernetes.io/projected/948e2234-c69a-47c6-b4f1-6d61d8ac19dd-kube-api-access-v7cpf\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:02:53.229335 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:53.229312 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/948e2234-c69a-47c6-b4f1-6d61d8ac19dd-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:02:53.276885 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:53.276855 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-46590df2-7d46fd668-6cc2h" Nov 03 21:02:53.330684 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:53.330649 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b7fadf9-8365-466e-925f-c4fb8ff49955-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4ad2ebc-hxffw\" (UID: \"6b7fadf9-8365-466e-925f-c4fb8ff49955\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ad2ebc-hxffw" Nov 03 21:02:53.330944 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:53.330901 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5mgk\" (UniqueName: \"kubernetes.io/projected/6b7fadf9-8365-466e-925f-c4fb8ff49955-kube-api-access-x5mgk\") pod \"maas-api-revoke-sub-d4ad2ebc-hxffw\" (UID: \"6b7fadf9-8365-466e-925f-c4fb8ff49955\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ad2ebc-hxffw" Nov 03 21:02:53.331077 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:53.330950 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b7fadf9-8365-466e-925f-c4fb8ff49955-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4ad2ebc-hxffw\" (UID: \"6b7fadf9-8365-466e-925f-c4fb8ff49955\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ad2ebc-hxffw" Nov 03 21:02:53.331792 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:53.331771 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b7fadf9-8365-466e-925f-c4fb8ff49955-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4ad2ebc-hxffw\" (UID: \"6b7fadf9-8365-466e-925f-c4fb8ff49955\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ad2ebc-hxffw" Nov 03 21:02:53.333255 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:53.333236 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b7fadf9-8365-466e-925f-c4fb8ff49955-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4ad2ebc-hxffw\" (UID: \"6b7fadf9-8365-466e-925f-c4fb8ff49955\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ad2ebc-hxffw" Nov 03 21:02:53.396673 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:53.396635 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5mgk\" (UniqueName: \"kubernetes.io/projected/6b7fadf9-8365-466e-925f-c4fb8ff49955-kube-api-access-x5mgk\") pod \"maas-api-revoke-sub-d4ad2ebc-hxffw\" (UID: \"6b7fadf9-8365-466e-925f-c4fb8ff49955\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ad2ebc-hxffw" Nov 03 21:02:53.472289 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:53.472203 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ad2ebc-hxffw" Nov 03 21:02:53.890160 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:02:53.890119 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b7fadf9_8365_466e_925f_c4fb8ff49955.slice/crio-3f16cb8b07895ca35788b28a9df9c44bc54ecd6245a4b53515bb9dc448db9c22 WatchSource:0}: Error finding container 3f16cb8b07895ca35788b28a9df9c44bc54ecd6245a4b53515bb9dc448db9c22: Status 404 returned error can't find the container with id 3f16cb8b07895ca35788b28a9df9c44bc54ecd6245a4b53515bb9dc448db9c22 Nov 03 21:02:53.891530 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:53.891493 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4ad2ebc-hxffw"] Nov 03 21:02:53.940364 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:53.940300 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ad2ebc-hxffw" event={"ID":"6b7fadf9-8365-466e-925f-c4fb8ff49955","Type":"ContainerStarted","Data":"3f16cb8b07895ca35788b28a9df9c44bc54ecd6245a4b53515bb9dc448db9c22"} Nov 03 21:02:53.942946 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:53.942924 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d830196-vhlrr" Nov 03 21:02:53.943057 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:53.942921 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d830196-vhlrr" event={"ID":"948e2234-c69a-47c6-b4f1-6d61d8ac19dd","Type":"ContainerDied","Data":"78c7b43afd6374507696fcfd93f0f58ead5329473fe19fbf5786c9059a810a8e"} Nov 03 21:02:53.943131 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:53.943095 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78c7b43afd6374507696fcfd93f0f58ead5329473fe19fbf5786c9059a810a8e" Nov 03 21:02:54.414166 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:54.414133 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-46590df2-66f54d6b9f-b48rg" Nov 03 21:02:54.953135 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:54.952949 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ad2ebc-hxffw" event={"ID":"6b7fadf9-8365-466e-925f-c4fb8ff49955","Type":"ContainerStarted","Data":"0701da96404727ac9cb151aadbc88365711690aa75fdfd45676d67aafd67386f"} Nov 03 21:02:55.076877 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:55.076812 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ad2ebc-hxffw" podStartSLOduration=2.076795525 podStartE2EDuration="2.076795525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:02:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:02:55.07065462 +0000 UTC m=+1410.295974426" watchObservedRunningTime="2026-11-03 21:02:55.076795525 +0000 UTC m=+1410.302115315" Nov 03 21:02:55.963309 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:55.963273 4305 generic.go:356] "Generic (PLEG): container finished" podID="6b7fadf9-8365-466e-925f-c4fb8ff49955" containerID="0701da96404727ac9cb151aadbc88365711690aa75fdfd45676d67aafd67386f" exitCode=0 Nov 03 21:02:55.963817 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:55.963359 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ad2ebc-hxffw" event={"ID":"6b7fadf9-8365-466e-925f-c4fb8ff49955","Type":"ContainerDied","Data":"0701da96404727ac9cb151aadbc88365711690aa75fdfd45676d67aafd67386f"} Nov 03 21:02:57.151664 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:57.151636 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ad2ebc-hxffw" Nov 03 21:02:57.196198 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:57.196166 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b7fadf9-8365-466e-925f-c4fb8ff49955-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b7fadf9-8365-466e-925f-c4fb8ff49955-maas-api-openshift-ca\") pod \"6b7fadf9-8365-466e-925f-c4fb8ff49955\" (UID: \"6b7fadf9-8365-466e-925f-c4fb8ff49955\") " Nov 03 21:02:57.196354 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:57.196223 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b7fadf9-8365-466e-925f-c4fb8ff49955-kube-api-access-x5mgk\" (UniqueName: \"kubernetes.io/projected/6b7fadf9-8365-466e-925f-c4fb8ff49955-kube-api-access-x5mgk\") pod \"6b7fadf9-8365-466e-925f-c4fb8ff49955\" (UID: \"6b7fadf9-8365-466e-925f-c4fb8ff49955\") " Nov 03 21:02:57.196354 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:57.196249 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b7fadf9-8365-466e-925f-c4fb8ff49955-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b7fadf9-8365-466e-925f-c4fb8ff49955-maas-api-xks-ca\") pod \"6b7fadf9-8365-466e-925f-c4fb8ff49955\" (UID: \"6b7fadf9-8365-466e-925f-c4fb8ff49955\") " Nov 03 21:02:57.196563 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:57.196537 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b7fadf9-8365-466e-925f-c4fb8ff49955-maas-api-openshift-ca" pod "6b7fadf9-8365-466e-925f-c4fb8ff49955" (UID: "6b7fadf9-8365-466e-925f-c4fb8ff49955"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:02:57.197188 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:57.197156 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b7fadf9-8365-466e-925f-c4fb8ff49955-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:02:57.198595 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:57.198573 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b7fadf9-8365-466e-925f-c4fb8ff49955-maas-api-xks-ca" pod "6b7fadf9-8365-466e-925f-c4fb8ff49955" (UID: "6b7fadf9-8365-466e-925f-c4fb8ff49955"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:02:57.198870 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:57.198843 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b7fadf9-8365-466e-925f-c4fb8ff49955-kube-api-access-x5mgk" pod "6b7fadf9-8365-466e-925f-c4fb8ff49955" (UID: "6b7fadf9-8365-466e-925f-c4fb8ff49955"). InnerVolumeSpecName "kube-api-access-x5mgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:02:57.298452 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:57.298356 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5mgk\" (UniqueName: \"kubernetes.io/projected/6b7fadf9-8365-466e-925f-c4fb8ff49955-kube-api-access-x5mgk\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:02:57.298452 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:57.298386 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b7fadf9-8365-466e-925f-c4fb8ff49955-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:02:57.983411 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:57.983372 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ad2ebc-hxffw" Nov 03 21:02:57.983411 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:57.983400 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ad2ebc-hxffw" event={"ID":"6b7fadf9-8365-466e-925f-c4fb8ff49955","Type":"ContainerDied","Data":"3f16cb8b07895ca35788b28a9df9c44bc54ecd6245a4b53515bb9dc448db9c22"} Nov 03 21:02:57.983631 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:02:57.983492 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f16cb8b07895ca35788b28a9df9c44bc54ecd6245a4b53515bb9dc448db9c22" Nov 03 21:03:05.800501 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:05.800447 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz"] Nov 03 21:03:05.801222 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:05.801205 4305 cpu_manager.go:414] "containerMap: removing container" podUID="6b7fadf9-8365-466e-925f-c4fb8ff49955" containerName="revoke-keys" Nov 03 21:03:05.801271 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:05.801224 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b7fadf9-8365-466e-925f-c4fb8ff49955" containerName="revoke-keys" Nov 03 21:03:05.801359 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:05.801350 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b7fadf9-8365-466e-925f-c4fb8ff49955" containerName="revoke-keys" Nov 03 21:03:05.806261 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:05.806242 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz" Nov 03 21:03:05.866134 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:05.865918 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-e3917193-openshift-default-dockercfg-dzvq4\"" Nov 03 21:03:05.923261 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:05.923228 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz"] Nov 03 21:03:06.014462 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:05.959585 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95dd3f17-e145-49ec-8fe2-6148c3f07c80-istiod-ca-cert\") pod \"e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz\" (UID: \"95dd3f17-e145-49ec-8fe2-6148c3f07c80\") " pod="openshift-ingress/e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz" Nov 03 21:03:06.066516 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:06.062022 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95dd3f17-e145-49ec-8fe2-6148c3f07c80-istio-data\") pod \"e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz\" (UID: \"95dd3f17-e145-49ec-8fe2-6148c3f07c80\") " pod="openshift-ingress/e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz" Nov 03 21:03:06.066516 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:06.062109 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95dd3f17-e145-49ec-8fe2-6148c3f07c80-credential-socket\") pod \"e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz\" (UID: \"95dd3f17-e145-49ec-8fe2-6148c3f07c80\") " pod="openshift-ingress/e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz" Nov 03 21:03:06.066516 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:06.062147 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95dd3f17-e145-49ec-8fe2-6148c3f07c80-workload-socket\") pod \"e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz\" (UID: \"95dd3f17-e145-49ec-8fe2-6148c3f07c80\") " pod="openshift-ingress/e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz" Nov 03 21:03:06.066516 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:06.062240 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95dd3f17-e145-49ec-8fe2-6148c3f07c80-istio-token\") pod \"e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz\" (UID: \"95dd3f17-e145-49ec-8fe2-6148c3f07c80\") " pod="openshift-ingress/e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz" Nov 03 21:03:06.066516 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:06.062303 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wvw5\" (UniqueName: \"kubernetes.io/projected/95dd3f17-e145-49ec-8fe2-6148c3f07c80-kube-api-access-8wvw5\") pod \"e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz\" (UID: \"95dd3f17-e145-49ec-8fe2-6148c3f07c80\") " pod="openshift-ingress/e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz" Nov 03 21:03:06.066516 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:06.062401 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95dd3f17-e145-49ec-8fe2-6148c3f07c80-workload-certs\") pod \"e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz\" (UID: \"95dd3f17-e145-49ec-8fe2-6148c3f07c80\") " pod="openshift-ingress/e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz" Nov 03 21:03:06.066516 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:06.064524 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95dd3f17-e145-49ec-8fe2-6148c3f07c80-istio-envoy\") pod \"e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz\" (UID: \"95dd3f17-e145-49ec-8fe2-6148c3f07c80\") " pod="openshift-ingress/e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz" Nov 03 21:03:06.066516 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:06.064657 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95dd3f17-e145-49ec-8fe2-6148c3f07c80-istio-podinfo\") pod \"e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz\" (UID: \"95dd3f17-e145-49ec-8fe2-6148c3f07c80\") " pod="openshift-ingress/e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz" Nov 03 21:03:06.166500 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:06.166236 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95dd3f17-e145-49ec-8fe2-6148c3f07c80-istio-data\") pod \"e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz\" (UID: \"95dd3f17-e145-49ec-8fe2-6148c3f07c80\") " pod="openshift-ingress/e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz" Nov 03 21:03:06.166500 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:06.166294 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95dd3f17-e145-49ec-8fe2-6148c3f07c80-credential-socket\") pod \"e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz\" (UID: \"95dd3f17-e145-49ec-8fe2-6148c3f07c80\") " pod="openshift-ingress/e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz" Nov 03 21:03:06.166500 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:06.166321 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95dd3f17-e145-49ec-8fe2-6148c3f07c80-workload-socket\") pod \"e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz\" (UID: \"95dd3f17-e145-49ec-8fe2-6148c3f07c80\") " pod="openshift-ingress/e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz" Nov 03 21:03:06.166500 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:06.166372 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95dd3f17-e145-49ec-8fe2-6148c3f07c80-istio-token\") pod \"e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz\" (UID: \"95dd3f17-e145-49ec-8fe2-6148c3f07c80\") " pod="openshift-ingress/e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz" Nov 03 21:03:06.166500 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:06.166423 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wvw5\" (UniqueName: \"kubernetes.io/projected/95dd3f17-e145-49ec-8fe2-6148c3f07c80-kube-api-access-8wvw5\") pod \"e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz\" (UID: \"95dd3f17-e145-49ec-8fe2-6148c3f07c80\") " pod="openshift-ingress/e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz" Nov 03 21:03:06.166907 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:06.166546 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95dd3f17-e145-49ec-8fe2-6148c3f07c80-workload-certs\") pod \"e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz\" (UID: \"95dd3f17-e145-49ec-8fe2-6148c3f07c80\") " pod="openshift-ingress/e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz" Nov 03 21:03:06.166907 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:06.166626 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95dd3f17-e145-49ec-8fe2-6148c3f07c80-istio-envoy\") pod \"e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz\" (UID: \"95dd3f17-e145-49ec-8fe2-6148c3f07c80\") " pod="openshift-ingress/e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz" Nov 03 21:03:06.166907 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:06.166675 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95dd3f17-e145-49ec-8fe2-6148c3f07c80-istio-podinfo\") pod \"e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz\" (UID: \"95dd3f17-e145-49ec-8fe2-6148c3f07c80\") " pod="openshift-ingress/e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz" Nov 03 21:03:06.166907 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:06.166770 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95dd3f17-e145-49ec-8fe2-6148c3f07c80-istiod-ca-cert\") pod \"e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz\" (UID: \"95dd3f17-e145-49ec-8fe2-6148c3f07c80\") " pod="openshift-ingress/e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz" Nov 03 21:03:06.167556 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:06.167526 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95dd3f17-e145-49ec-8fe2-6148c3f07c80-istiod-ca-cert\") pod \"e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz\" (UID: \"95dd3f17-e145-49ec-8fe2-6148c3f07c80\") " pod="openshift-ingress/e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz" Nov 03 21:03:06.168016 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:06.167970 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95dd3f17-e145-49ec-8fe2-6148c3f07c80-workload-socket\") pod \"e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz\" (UID: \"95dd3f17-e145-49ec-8fe2-6148c3f07c80\") " pod="openshift-ingress/e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz" Nov 03 21:03:06.168016 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:06.168010 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95dd3f17-e145-49ec-8fe2-6148c3f07c80-credential-socket\") pod \"e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz\" (UID: \"95dd3f17-e145-49ec-8fe2-6148c3f07c80\") " pod="openshift-ingress/e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz" Nov 03 21:03:06.168526 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:06.168503 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95dd3f17-e145-49ec-8fe2-6148c3f07c80-workload-certs\") pod \"e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz\" (UID: \"95dd3f17-e145-49ec-8fe2-6148c3f07c80\") " pod="openshift-ingress/e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz" Nov 03 21:03:06.168873 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:06.168853 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95dd3f17-e145-49ec-8fe2-6148c3f07c80-istio-data\") pod \"e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz\" (UID: \"95dd3f17-e145-49ec-8fe2-6148c3f07c80\") " pod="openshift-ingress/e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz" Nov 03 21:03:06.170355 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:06.170336 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95dd3f17-e145-49ec-8fe2-6148c3f07c80-istio-podinfo\") pod \"e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz\" (UID: \"95dd3f17-e145-49ec-8fe2-6148c3f07c80\") " pod="openshift-ingress/e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz" Nov 03 21:03:06.170501 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:06.170405 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95dd3f17-e145-49ec-8fe2-6148c3f07c80-istio-envoy\") pod \"e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz\" (UID: \"95dd3f17-e145-49ec-8fe2-6148c3f07c80\") " pod="openshift-ingress/e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz" Nov 03 21:03:06.238128 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:06.238051 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95dd3f17-e145-49ec-8fe2-6148c3f07c80-istio-token\") pod \"e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz\" (UID: \"95dd3f17-e145-49ec-8fe2-6148c3f07c80\") " pod="openshift-ingress/e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz" Nov 03 21:03:06.238128 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:06.238095 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wvw5\" (UniqueName: \"kubernetes.io/projected/95dd3f17-e145-49ec-8fe2-6148c3f07c80-kube-api-access-8wvw5\") pod \"e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz\" (UID: \"95dd3f17-e145-49ec-8fe2-6148c3f07c80\") " pod="openshift-ingress/e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz" Nov 03 21:03:06.423050 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:06.421298 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz" Nov 03 21:03:06.883114 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:03:06.883082 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95dd3f17_e145_49ec_8fe2_6148c3f07c80.slice/crio-7e03790f025b0417c6e4f29adef4d514bb01facba5538d293177946e16b193d4 WatchSource:0}: Error finding container 7e03790f025b0417c6e4f29adef4d514bb01facba5538d293177946e16b193d4: Status 404 returned error can't find the container with id 7e03790f025b0417c6e4f29adef4d514bb01facba5538d293177946e16b193d4 Nov 03 21:03:06.883567 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:06.883541 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz"] Nov 03 21:03:06.886687 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:06.886654 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:03:06.886808 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:06.886726 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:03:06.886808 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:06.886755 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:03:07.163290 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:07.163239 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz" event={"ID":"95dd3f17-e145-49ec-8fe2-6148c3f07c80","Type":"ContainerStarted","Data":"18c6ebd03bc3927d7424bc82cfe496702ce416880bdeeaa6523167f626cbe951"} Nov 03 21:03:07.163492 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:07.163326 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz" event={"ID":"95dd3f17-e145-49ec-8fe2-6148c3f07c80","Type":"ContainerStarted","Data":"7e03790f025b0417c6e4f29adef4d514bb01facba5538d293177946e16b193d4"} Nov 03 21:03:07.290500 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:07.290418 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz" podStartSLOduration=2.290398206 podStartE2EDuration="2.290398206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:03:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:03:07.288244728 +0000 UTC m=+1422.513564540" watchObservedRunningTime="2026-11-03 21:03:07.290398206 +0000 UTC m=+1422.515717997" Nov 03 21:03:07.424540 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:07.422412 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz" Nov 03 21:03:07.434397 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:07.434356 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz" Nov 03 21:03:08.172153 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:08.172116 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz" Nov 03 21:03:08.173346 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:08.173326 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz" Nov 03 21:03:14.337519 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:14.337484 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-6f2c17e9-75c5d4f87f-j6p4t"] Nov 03 21:03:14.343076 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:14.343047 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-6f2c17e9-75c5d4f87f-j6p4t" Nov 03 21:03:14.403145 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:14.403107 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-6f2c17e9-75c5d4f87f-j6p4t"] Nov 03 21:03:14.455534 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:14.452970 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-6f2c17e9\"" Nov 03 21:03:14.455534 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:14.453206 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-6f2c17e9-dockercfg-f6kgq\"" Nov 03 21:03:14.583192 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:14.574110 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f614f399-012f-43f3-bd18-cb62b5d089d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-6f2c17e9-75c5d4f87f-j6p4t\" (UID: \"f614f399-012f-43f3-bd18-cb62b5d089d7\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-6f2c17e9-75c5d4f87f-j6p4t" Nov 03 21:03:14.583192 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:14.574383 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzs6d\" (UniqueName: \"kubernetes.io/projected/f614f399-012f-43f3-bd18-cb62b5d089d7-kube-api-access-wzs6d\") pod \"payload-pre-processing-e2e-derive-6f2c17e9-75c5d4f87f-j6p4t\" (UID: \"f614f399-012f-43f3-bd18-cb62b5d089d7\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-6f2c17e9-75c5d4f87f-j6p4t" Nov 03 21:03:14.649559 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:14.649524 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-6f2c17e9-747c59f698-dkf65"] Nov 03 21:03:14.654479 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:14.654453 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-6f2c17e9-747c59f698-dkf65" Nov 03 21:03:14.656591 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:14.656563 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-6f2c17e9-747c59f698-dkf65"] Nov 03 21:03:14.676320 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:14.676275 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f614f399-012f-43f3-bd18-cb62b5d089d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-6f2c17e9-75c5d4f87f-j6p4t\" (UID: \"f614f399-012f-43f3-bd18-cb62b5d089d7\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-6f2c17e9-75c5d4f87f-j6p4t" Nov 03 21:03:14.677307 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:14.677281 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzs6d\" (UniqueName: \"kubernetes.io/projected/f614f399-012f-43f3-bd18-cb62b5d089d7-kube-api-access-wzs6d\") pod \"payload-pre-processing-e2e-derive-6f2c17e9-75c5d4f87f-j6p4t\" (UID: \"f614f399-012f-43f3-bd18-cb62b5d089d7\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-6f2c17e9-75c5d4f87f-j6p4t" Nov 03 21:03:14.678204 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:14.678184 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f614f399-012f-43f3-bd18-cb62b5d089d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-6f2c17e9-75c5d4f87f-j6p4t\" (UID: \"f614f399-012f-43f3-bd18-cb62b5d089d7\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-6f2c17e9-75c5d4f87f-j6p4t" Nov 03 21:03:14.738523 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:14.738481 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzs6d\" (UniqueName: \"kubernetes.io/projected/f614f399-012f-43f3-bd18-cb62b5d089d7-kube-api-access-wzs6d\") pod \"payload-pre-processing-e2e-derive-6f2c17e9-75c5d4f87f-j6p4t\" (UID: \"f614f399-012f-43f3-bd18-cb62b5d089d7\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-6f2c17e9-75c5d4f87f-j6p4t" Nov 03 21:03:14.779258 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:14.779221 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d6mc\" (UniqueName: \"kubernetes.io/projected/1b9f328c-f07b-4b97-ab85-0b94cbd3676c-kube-api-access-9d6mc\") pod \"payload-processing-e2e-derive-6f2c17e9-747c59f698-dkf65\" (UID: \"1b9f328c-f07b-4b97-ab85-0b94cbd3676c\") " pod="openshift-ingress/payload-processing-e2e-derive-6f2c17e9-747c59f698-dkf65" Nov 03 21:03:14.779559 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:14.779538 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b9f328c-f07b-4b97-ab85-0b94cbd3676c-plugins-config-volume\") pod \"payload-processing-e2e-derive-6f2c17e9-747c59f698-dkf65\" (UID: \"1b9f328c-f07b-4b97-ab85-0b94cbd3676c\") " pod="openshift-ingress/payload-processing-e2e-derive-6f2c17e9-747c59f698-dkf65" Nov 03 21:03:14.881375 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:14.881336 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d6mc\" (UniqueName: \"kubernetes.io/projected/1b9f328c-f07b-4b97-ab85-0b94cbd3676c-kube-api-access-9d6mc\") pod \"payload-processing-e2e-derive-6f2c17e9-747c59f698-dkf65\" (UID: \"1b9f328c-f07b-4b97-ab85-0b94cbd3676c\") " pod="openshift-ingress/payload-processing-e2e-derive-6f2c17e9-747c59f698-dkf65" Nov 03 21:03:14.881667 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:14.881649 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b9f328c-f07b-4b97-ab85-0b94cbd3676c-plugins-config-volume\") pod \"payload-processing-e2e-derive-6f2c17e9-747c59f698-dkf65\" (UID: \"1b9f328c-f07b-4b97-ab85-0b94cbd3676c\") " pod="openshift-ingress/payload-processing-e2e-derive-6f2c17e9-747c59f698-dkf65" Nov 03 21:03:14.882360 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:14.882338 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b9f328c-f07b-4b97-ab85-0b94cbd3676c-plugins-config-volume\") pod \"payload-processing-e2e-derive-6f2c17e9-747c59f698-dkf65\" (UID: \"1b9f328c-f07b-4b97-ab85-0b94cbd3676c\") " pod="openshift-ingress/payload-processing-e2e-derive-6f2c17e9-747c59f698-dkf65" Nov 03 21:03:14.942305 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:14.942224 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d6mc\" (UniqueName: \"kubernetes.io/projected/1b9f328c-f07b-4b97-ab85-0b94cbd3676c-kube-api-access-9d6mc\") pod \"payload-processing-e2e-derive-6f2c17e9-747c59f698-dkf65\" (UID: \"1b9f328c-f07b-4b97-ab85-0b94cbd3676c\") " pod="openshift-ingress/payload-processing-e2e-derive-6f2c17e9-747c59f698-dkf65" Nov 03 21:03:14.960331 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:14.960303 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-6f2c17e9-75c5d4f87f-j6p4t" Nov 03 21:03:14.974991 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:14.974949 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-6f2c17e9-747c59f698-dkf65" Nov 03 21:03:15.355508 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:15.355322 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-6f2c17e9-75c5d4f87f-j6p4t"] Nov 03 21:03:15.390898 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:03:15.390864 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf614f399_012f_43f3_bd18_cb62b5d089d7.slice/crio-6481b0004369c8e9487891df03d9e5e2712e4b80595236d3b70fde62f8f43fa9 WatchSource:0}: Error finding container 6481b0004369c8e9487891df03d9e5e2712e4b80595236d3b70fde62f8f43fa9: Status 404 returned error can't find the container with id 6481b0004369c8e9487891df03d9e5e2712e4b80595236d3b70fde62f8f43fa9 Nov 03 21:03:15.405051 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:15.405029 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-6f2c17e9-747c59f698-dkf65"] Nov 03 21:03:15.407241 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:03:15.407214 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b9f328c_f07b_4b97_ab85_0b94cbd3676c.slice/crio-89560cffc0f2781fcd6b56da7f60463f4727c315f22e9bc20ae23ca264da1ea5 WatchSource:0}: Error finding container 89560cffc0f2781fcd6b56da7f60463f4727c315f22e9bc20ae23ca264da1ea5: Status 404 returned error can't find the container with id 89560cffc0f2781fcd6b56da7f60463f4727c315f22e9bc20ae23ca264da1ea5 Nov 03 21:03:16.246180 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:16.246145 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-6f2c17e9-747c59f698-dkf65" event={"ID":"1b9f328c-f07b-4b97-ab85-0b94cbd3676c","Type":"ContainerStarted","Data":"bb550ff03bf04711ca7e01d6361e40e0ed6404561cc07dc66022dbc6cee88cdf"} Nov 03 21:03:16.246409 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:16.246199 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-6f2c17e9-747c59f698-dkf65" event={"ID":"1b9f328c-f07b-4b97-ab85-0b94cbd3676c","Type":"ContainerStarted","Data":"89560cffc0f2781fcd6b56da7f60463f4727c315f22e9bc20ae23ca264da1ea5"} Nov 03 21:03:16.246409 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:16.246240 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-6f2c17e9-747c59f698-dkf65" Nov 03 21:03:16.248130 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:16.248103 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-6f2c17e9-75c5d4f87f-j6p4t" event={"ID":"f614f399-012f-43f3-bd18-cb62b5d089d7","Type":"ContainerStarted","Data":"512c1ccd3f2dfd7cc3ca159509480d6c9486d0524dd10abd71dc50e50dd08062"} Nov 03 21:03:16.248251 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:16.248148 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-6f2c17e9-75c5d4f87f-j6p4t" event={"ID":"f614f399-012f-43f3-bd18-cb62b5d089d7","Type":"ContainerStarted","Data":"6481b0004369c8e9487891df03d9e5e2712e4b80595236d3b70fde62f8f43fa9"} Nov 03 21:03:16.248251 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:16.248169 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-6f2c17e9-75c5d4f87f-j6p4t" Nov 03 21:03:16.325362 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:03:16.325330 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:03:16.366508 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:16.366447 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-6f2c17e9-747c59f698-dkf65" podStartSLOduration=2.366413471 podStartE2EDuration="2.366413471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:03:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:03:16.362403488 +0000 UTC m=+1431.587723300" watchObservedRunningTime="2026-11-03 21:03:16.366413471 +0000 UTC m=+1431.591733260" Nov 03 21:03:16.485216 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:16.485148 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-6f2c17e9-75c5d4f87f-j6p4t" podStartSLOduration=2.485124859 podStartE2EDuration="2.485124859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:03:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:03:16.48125342 +0000 UTC m=+1431.706573214" watchObservedRunningTime="2026-11-03 21:03:16.485124859 +0000 UTC m=+1431.710444651" Nov 03 21:03:21.748660 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:21.748619 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e3917193-665777fcf9-khvcd"] Nov 03 21:03:21.768516 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:21.758349 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e3917193-665777fcf9-khvcd" Nov 03 21:03:21.840171 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:21.840131 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e3917193-665777fcf9-khvcd"] Nov 03 21:03:21.841329 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:21.840750 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-e3917193\"" Nov 03 21:03:21.841329 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:21.840995 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-e3917193-dockercfg-ncn96\"" Nov 03 21:03:21.884129 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:21.884090 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e3917193-68c654f4dd-ttdqc"] Nov 03 21:03:21.894235 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:21.894202 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e3917193-68c654f4dd-ttdqc" Nov 03 21:03:21.942065 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:21.942017 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg4ml\" (UniqueName: \"kubernetes.io/projected/998a67c4-3f64-4922-a2c6-4b7629c96b28-kube-api-access-mg4ml\") pod \"payload-pre-processing-e2e-mt-e3917193-665777fcf9-khvcd\" (UID: \"998a67c4-3f64-4922-a2c6-4b7629c96b28\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e3917193-665777fcf9-khvcd" Nov 03 21:03:21.942247 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:21.942212 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/998a67c4-3f64-4922-a2c6-4b7629c96b28-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e3917193-665777fcf9-khvcd\" (UID: \"998a67c4-3f64-4922-a2c6-4b7629c96b28\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e3917193-665777fcf9-khvcd" Nov 03 21:03:21.943058 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:21.943034 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e3917193-68c654f4dd-ttdqc"] Nov 03 21:03:22.067856 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:22.064077 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkgcr\" (UniqueName: \"kubernetes.io/projected/facfcb94-b6b7-4072-b4f0-49ef05e5dde8-kube-api-access-tkgcr\") pod \"payload-processing-e2e-mt-e3917193-68c654f4dd-ttdqc\" (UID: \"facfcb94-b6b7-4072-b4f0-49ef05e5dde8\") " pod="openshift-ingress/payload-processing-e2e-mt-e3917193-68c654f4dd-ttdqc" Nov 03 21:03:22.067856 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:22.064271 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg4ml\" (UniqueName: \"kubernetes.io/projected/998a67c4-3f64-4922-a2c6-4b7629c96b28-kube-api-access-mg4ml\") pod \"payload-pre-processing-e2e-mt-e3917193-665777fcf9-khvcd\" (UID: \"998a67c4-3f64-4922-a2c6-4b7629c96b28\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e3917193-665777fcf9-khvcd" Nov 03 21:03:22.067856 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:22.064403 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/998a67c4-3f64-4922-a2c6-4b7629c96b28-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e3917193-665777fcf9-khvcd\" (UID: \"998a67c4-3f64-4922-a2c6-4b7629c96b28\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e3917193-665777fcf9-khvcd" Nov 03 21:03:22.067856 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:22.064548 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/facfcb94-b6b7-4072-b4f0-49ef05e5dde8-plugins-config-volume\") pod \"payload-processing-e2e-mt-e3917193-68c654f4dd-ttdqc\" (UID: \"facfcb94-b6b7-4072-b4f0-49ef05e5dde8\") " pod="openshift-ingress/payload-processing-e2e-mt-e3917193-68c654f4dd-ttdqc" Nov 03 21:03:22.067856 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:22.065861 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/998a67c4-3f64-4922-a2c6-4b7629c96b28-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e3917193-665777fcf9-khvcd\" (UID: \"998a67c4-3f64-4922-a2c6-4b7629c96b28\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e3917193-665777fcf9-khvcd" Nov 03 21:03:22.138494 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:22.135691 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg4ml\" (UniqueName: \"kubernetes.io/projected/998a67c4-3f64-4922-a2c6-4b7629c96b28-kube-api-access-mg4ml\") pod \"payload-pre-processing-e2e-mt-e3917193-665777fcf9-khvcd\" (UID: \"998a67c4-3f64-4922-a2c6-4b7629c96b28\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e3917193-665777fcf9-khvcd" Nov 03 21:03:22.166633 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:22.166586 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/facfcb94-b6b7-4072-b4f0-49ef05e5dde8-plugins-config-volume\") pod \"payload-processing-e2e-mt-e3917193-68c654f4dd-ttdqc\" (UID: \"facfcb94-b6b7-4072-b4f0-49ef05e5dde8\") " pod="openshift-ingress/payload-processing-e2e-mt-e3917193-68c654f4dd-ttdqc" Nov 03 21:03:22.166806 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:22.166738 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkgcr\" (UniqueName: \"kubernetes.io/projected/facfcb94-b6b7-4072-b4f0-49ef05e5dde8-kube-api-access-tkgcr\") pod \"payload-processing-e2e-mt-e3917193-68c654f4dd-ttdqc\" (UID: \"facfcb94-b6b7-4072-b4f0-49ef05e5dde8\") " pod="openshift-ingress/payload-processing-e2e-mt-e3917193-68c654f4dd-ttdqc" Nov 03 21:03:22.168539 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:22.168509 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/facfcb94-b6b7-4072-b4f0-49ef05e5dde8-plugins-config-volume\") pod \"payload-processing-e2e-mt-e3917193-68c654f4dd-ttdqc\" (UID: \"facfcb94-b6b7-4072-b4f0-49ef05e5dde8\") " pod="openshift-ingress/payload-processing-e2e-mt-e3917193-68c654f4dd-ttdqc" Nov 03 21:03:22.232547 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:22.232511 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkgcr\" (UniqueName: \"kubernetes.io/projected/facfcb94-b6b7-4072-b4f0-49ef05e5dde8-kube-api-access-tkgcr\") pod \"payload-processing-e2e-mt-e3917193-68c654f4dd-ttdqc\" (UID: \"facfcb94-b6b7-4072-b4f0-49ef05e5dde8\") " pod="openshift-ingress/payload-processing-e2e-mt-e3917193-68c654f4dd-ttdqc" Nov 03 21:03:22.417215 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:22.417182 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e3917193-665777fcf9-khvcd" Nov 03 21:03:22.520498 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:22.520465 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e3917193-68c654f4dd-ttdqc" Nov 03 21:03:22.814318 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:22.814277 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e3917193-665777fcf9-khvcd"] Nov 03 21:03:22.816460 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:03:22.816397 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod998a67c4_3f64_4922_a2c6_4b7629c96b28.slice/crio-a99795044dded5fe0a9848791d2de1fe1664ff8cfd950c9370aaa675447a4962 WatchSource:0}: Error finding container a99795044dded5fe0a9848791d2de1fe1664ff8cfd950c9370aaa675447a4962: Status 404 returned error can't find the container with id a99795044dded5fe0a9848791d2de1fe1664ff8cfd950c9370aaa675447a4962 Nov 03 21:03:22.920701 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:03:22.920672 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfacfcb94_b6b7_4072_b4f0_49ef05e5dde8.slice/crio-58eec6978b0b123f990592be7f586680a451025861e25fcc9954b42ea9a2e6cf WatchSource:0}: Error finding container 58eec6978b0b123f990592be7f586680a451025861e25fcc9954b42ea9a2e6cf: Status 404 returned error can't find the container with id 58eec6978b0b123f990592be7f586680a451025861e25fcc9954b42ea9a2e6cf Nov 03 21:03:22.921648 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:22.921623 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e3917193-68c654f4dd-ttdqc"] Nov 03 21:03:23.315588 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:23.315555 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e3917193-68c654f4dd-ttdqc" event={"ID":"facfcb94-b6b7-4072-b4f0-49ef05e5dde8","Type":"ContainerStarted","Data":"20d35ec6d07ff7fb4194128f42ebbc5cc3bce3071306d400ab7770409f9c194e"} Nov 03 21:03:23.315588 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:23.315608 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e3917193-68c654f4dd-ttdqc" event={"ID":"facfcb94-b6b7-4072-b4f0-49ef05e5dde8","Type":"ContainerStarted","Data":"58eec6978b0b123f990592be7f586680a451025861e25fcc9954b42ea9a2e6cf"} Nov 03 21:03:23.315875 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:23.315632 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-e3917193-68c654f4dd-ttdqc" Nov 03 21:03:23.317303 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:23.317274 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e3917193-665777fcf9-khvcd" event={"ID":"998a67c4-3f64-4922-a2c6-4b7629c96b28","Type":"ContainerStarted","Data":"44c8a7d9abe1fbe91326914c43322962425f9f303191432c48298c80cfa54134"} Nov 03 21:03:23.317460 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:23.317325 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e3917193-665777fcf9-khvcd" event={"ID":"998a67c4-3f64-4922-a2c6-4b7629c96b28","Type":"ContainerStarted","Data":"a99795044dded5fe0a9848791d2de1fe1664ff8cfd950c9370aaa675447a4962"} Nov 03 21:03:23.317460 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:23.317359 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e3917193-665777fcf9-khvcd" Nov 03 21:03:23.431473 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:23.431326 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-e3917193-68c654f4dd-ttdqc" podStartSLOduration=2.431285386 podStartE2EDuration="2.431285386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:03:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:03:23.430970174 +0000 UTC m=+1438.656289990" watchObservedRunningTime="2026-11-03 21:03:23.431285386 +0000 UTC m=+1438.656605178" Nov 03 21:03:23.547374 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:23.547297 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-e3917193-665777fcf9-khvcd" podStartSLOduration=2.547269908 podStartE2EDuration="2.547269908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:03:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:03:23.545683419 +0000 UTC m=+1438.771003211" watchObservedRunningTime="2026-11-03 21:03:23.547269908 +0000 UTC m=+1438.772589716" Nov 03 21:03:27.262320 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:27.262288 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-6f2c17e9-75c5d4f87f-j6p4t" Nov 03 21:03:27.262793 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:27.262373 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-6f2c17e9-747c59f698-dkf65" Nov 03 21:03:29.523171 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:29.523135 4305 scope.go:122] "RemoveContainer" containerID="57dcd8efd6ddc3e473dcca0aab985606acb89473104660948b1f1f19cea2900e" Nov 03 21:03:29.535363 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:29.535336 4305 scope.go:122] "RemoveContainer" containerID="815381d59aab0887eb76d120a2f5d5d78c67c6b3fd29a590c66220fff2b0e9bb" Nov 03 21:03:29.554007 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:29.553983 4305 scope.go:122] "RemoveContainer" containerID="2a24fb82711a099fdd03fa72b356400287ad62cf8920375622648f1c47c5b731" Nov 03 21:03:29.565591 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:29.565571 4305 scope.go:122] "RemoveContainer" containerID="f666760b40b1db44c3ca5931742d2edab5a22e3c462266254e313e4f7181c8dd" Nov 03 21:03:29.577296 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:29.577274 4305 scope.go:122] "RemoveContainer" containerID="4e35ffdeab57a1483b3e9c61515f0b5ab2791da4589c005380386852f0e6b067" Nov 03 21:03:29.589036 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:29.589016 4305 scope.go:122] "RemoveContainer" containerID="ffca4bda878a750313991f2567c01b70d82e86bbccc678b68fe0b30d264c2b37" Nov 03 21:03:29.602401 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:29.602379 4305 scope.go:122] "RemoveContainer" containerID="9c4ef73d6ff73da4f421f0bff7c7fc214b45f6dff6cb0c5b966d3a38bffb3888" Nov 03 21:03:29.614334 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:29.614305 4305 scope.go:122] "RemoveContainer" containerID="453723776d937db36be3b89f44a51b136ec711c3ac982f8ef384e203e93358bf" Nov 03 21:03:29.626096 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:29.626070 4305 scope.go:122] "RemoveContainer" containerID="a3337c5cae7ab4b379111ff3148e80cb85402233c117e1eff1e03cd1b2070424" Nov 03 21:03:29.637278 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:29.637258 4305 scope.go:122] "RemoveContainer" containerID="e094719213d34a0e3d223891432f1a86cc1bbfdea6c2401700a41d3279e4213b" Nov 03 21:03:32.144511 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:32.144467 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" podUID="be06bc96-808a-46e6-9872-01baf1a11a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:03:32.270515 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:32.268538 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" podUID="8ee1f028-8503-4034-9954-eb06fcf402e7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:03:32.315982 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:32.315939 4305 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" podUID="d5d3dcaa-a51a-40c6-8049-871b300552e1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/health\": context deadline exceeded" Nov 03 21:03:32.505654 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:32.505543 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" podUID="63a704b4-51ce-46c3-8007-34b308aff332" containerName="main" probeResult="failure" output="Get \"https://10.134.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:03:32.507523 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:32.507489 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" podUID="7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:03:32.508030 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:32.507505 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" podUID="d5d3dcaa-a51a-40c6-8049-871b300552e1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:03:32.622379 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:32.621539 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" podUID="793b8ca3-3a2e-49a1-9be4-4971ba59df09" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 03 21:03:32.685476 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:32.676570 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" podUID="006c5eae-06af-447b-a1d8-cd719cce2a92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:03:32.738461 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:32.724695 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" podUID="6e7f30a9-0862-48a5-9ab7-e692e75bb294" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 03 21:03:33.115462 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:33.104694 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" podUID="7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:03:33.156652 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:33.156591 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" podUID="be06bc96-808a-46e6-9872-01baf1a11a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.107:8000/ready\": context deadline exceeded" Nov 03 21:03:33.157860 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:33.157834 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" Nov 03 21:03:33.269598 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:33.269542 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" podUID="8ee1f028-8503-4034-9954-eb06fcf402e7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:03:33.270558 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:33.270532 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" Nov 03 21:03:33.569585 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:33.538264 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" podUID="d5d3dcaa-a51a-40c6-8049-871b300552e1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:03:33.569585 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:33.538626 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" podUID="63a704b4-51ce-46c3-8007-34b308aff332" containerName="main" probeResult="failure" output="Get \"https://10.134.0.101:8000/ready\": context deadline exceeded" Nov 03 21:03:33.569585 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:33.542682 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" podUID="7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 03 21:03:33.569585 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:33.544027 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" Nov 03 21:03:33.569585 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:33.544123 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" Nov 03 21:03:33.569585 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:33.544161 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" Nov 03 21:03:33.626027 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:33.625986 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" podUID="793b8ca3-3a2e-49a1-9be4-4971ba59df09" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:03:33.640466 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:33.626715 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" Nov 03 21:03:33.788479 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:33.760130 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" podUID="006c5eae-06af-447b-a1d8-cd719cce2a92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded" Nov 03 21:03:33.788479 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:33.760562 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" Nov 03 21:03:34.076463 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:34.034152 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" podUID="6e7f30a9-0862-48a5-9ab7-e692e75bb294" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 03 21:03:34.076463 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:34.035703 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" Nov 03 21:03:34.127716 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:34.127259 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" podUID="8793f095-97b1-4e0a-8382-b541f90a0de2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 03 21:03:34.130974 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:34.130933 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" podUID="7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:03:34.146168 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:34.146135 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" Nov 03 21:03:34.159935 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:34.159876 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" podUID="be06bc96-808a-46e6-9872-01baf1a11a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:03:34.275595 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:34.275545 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" podUID="8ee1f028-8503-4034-9954-eb06fcf402e7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": context deadline exceeded" Nov 03 21:03:34.314587 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:34.314538 4305 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" podUID="7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:03:34.551699 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:34.542259 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" podUID="63a704b4-51ce-46c3-8007-34b308aff332" containerName="main" probeResult="failure" output="Get \"https://10.134.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:03:34.551699 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:34.542678 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" podUID="d5d3dcaa-a51a-40c6-8049-871b300552e1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded" Nov 03 21:03:34.551699 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:34.543620 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e3917193-665777fcf9-khvcd" Nov 03 21:03:34.551699 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:34.544258 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" podUID="7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:03:34.553813 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:34.553788 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-e3917193-68c654f4dd-ttdqc" Nov 03 21:03:34.670465 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:34.662539 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" podUID="793b8ca3-3a2e-49a1-9be4-4971ba59df09" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 03 21:03:34.803521 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:34.802536 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" podUID="80de7944-70f6-49e1-a0e8-4b82d5caad0d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:03:34.825463 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:34.812112 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" podUID="006c5eae-06af-447b-a1d8-cd719cce2a92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:03:35.059517 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:35.048515 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" podUID="6e7f30a9-0862-48a5-9ab7-e692e75bb294" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 03 21:03:35.240956 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:35.240904 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" podUID="8793f095-97b1-4e0a-8382-b541f90a0de2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 03 21:03:35.242155 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:35.242118 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" podUID="be06bc96-808a-46e6-9872-01baf1a11a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:03:35.467391 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:35.284031 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" podUID="8ee1f028-8503-4034-9954-eb06fcf402e7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": context deadline exceeded" Nov 03 21:03:35.467391 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:35.285161 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" podUID="7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:03:35.553890 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:35.497890 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" Nov 03 21:03:35.554275 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:35.554226 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c9n6g" podUID="5e264222-b7cf-4f08-85ff-f274be9562b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Nov 03 21:03:35.656216 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:35.656173 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" podUID="7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:03:35.889522 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:35.884459 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" podUID="793b8ca3-3a2e-49a1-9be4-4971ba59df09" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 03 21:03:35.932400 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:35.932358 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" podUID="006c5eae-06af-447b-a1d8-cd719cce2a92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:03:35.953200 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:35.953154 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" podUID="d5d3dcaa-a51a-40c6-8049-871b300552e1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded" Nov 03 21:03:35.955284 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:35.955211 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" podUID="80de7944-70f6-49e1-a0e8-4b82d5caad0d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Nov 03 21:03:36.180322 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:36.041836 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" Nov 03 21:03:36.180702 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:36.180655 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" Nov 03 21:03:36.182270 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:36.091938 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" podUID="63a704b4-51ce-46c3-8007-34b308aff332" containerName="main" probeResult="failure" output="Get \"https://10.134.0.101:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:03:36.190462 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:36.188529 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" Nov 03 21:03:36.194805 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:36.194780 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" Nov 03 21:03:36.227421 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:36.227392 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" Nov 03 21:03:36.242333 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:36.129930 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" podUID="6e7f30a9-0862-48a5-9ab7-e692e75bb294" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 03 21:03:36.245004 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:36.244972 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" podUID="8793f095-97b1-4e0a-8382-b541f90a0de2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 03 21:03:36.265476 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:36.245986 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" Nov 03 21:03:36.266062 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:36.265946 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" Nov 03 21:03:36.310760 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:36.310726 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" Nov 03 21:03:36.318685 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:36.315694 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" Nov 03 21:03:36.322260 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:36.322233 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" Nov 03 21:03:36.337734 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:36.335891 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" Nov 03 21:03:36.339682 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:36.339630 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" Nov 03 21:03:46.324767 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:03:46.324725 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:03:53.224076 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:53.224038 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-27ssq"] Nov 03 21:03:53.230421 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:53.230396 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-27ssq" Nov 03 21:03:53.263769 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:53.263733 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f183a1c-3354-4637-891c-9bcbfb0daf67-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-27ssq\" (UID: \"8f183a1c-3354-4637-891c-9bcbfb0daf67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-27ssq" Nov 03 21:03:53.263769 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:53.263777 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f183a1c-3354-4637-891c-9bcbfb0daf67-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-27ssq\" (UID: \"8f183a1c-3354-4637-891c-9bcbfb0daf67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-27ssq" Nov 03 21:03:53.264028 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:53.263914 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw8sr\" (UniqueName: \"kubernetes.io/projected/8f183a1c-3354-4637-891c-9bcbfb0daf67-kube-api-access-sw8sr\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-27ssq\" (UID: \"8f183a1c-3354-4637-891c-9bcbfb0daf67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-27ssq" Nov 03 21:03:53.264028 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:53.263952 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f183a1c-3354-4637-891c-9bcbfb0daf67-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-27ssq\" (UID: \"8f183a1c-3354-4637-891c-9bcbfb0daf67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-27ssq" Nov 03 21:03:53.264028 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:53.263972 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f183a1c-3354-4637-891c-9bcbfb0daf67-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-27ssq\" (UID: \"8f183a1c-3354-4637-891c-9bcbfb0daf67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-27ssq" Nov 03 21:03:53.264222 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:53.264130 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f183a1c-3354-4637-891c-9bcbfb0daf67-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-27ssq\" (UID: \"8f183a1c-3354-4637-891c-9bcbfb0daf67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-27ssq" Nov 03 21:03:53.264285 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:53.264227 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f183a1c-3354-4637-891c-9bcbfb0daf67-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-27ssq\" (UID: \"8f183a1c-3354-4637-891c-9bcbfb0daf67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-27ssq" Nov 03 21:03:53.264285 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:53.264263 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f183a1c-3354-4637-891c-9bcbfb0daf67-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-27ssq\" (UID: \"8f183a1c-3354-4637-891c-9bcbfb0daf67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-27ssq" Nov 03 21:03:53.264402 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:53.264381 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f183a1c-3354-4637-891c-9bcbfb0daf67-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-27ssq\" (UID: \"8f183a1c-3354-4637-891c-9bcbfb0daf67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-27ssq" Nov 03 21:03:53.327049 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:53.327012 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-27ssq"] Nov 03 21:03:53.366386 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:53.366344 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw8sr\" (UniqueName: \"kubernetes.io/projected/8f183a1c-3354-4637-891c-9bcbfb0daf67-kube-api-access-sw8sr\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-27ssq\" (UID: \"8f183a1c-3354-4637-891c-9bcbfb0daf67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-27ssq" Nov 03 21:03:53.366618 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:53.366475 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f183a1c-3354-4637-891c-9bcbfb0daf67-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-27ssq\" (UID: \"8f183a1c-3354-4637-891c-9bcbfb0daf67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-27ssq" Nov 03 21:03:53.366618 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:53.366519 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f183a1c-3354-4637-891c-9bcbfb0daf67-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-27ssq\" (UID: \"8f183a1c-3354-4637-891c-9bcbfb0daf67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-27ssq" Nov 03 21:03:53.366618 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:53.366579 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f183a1c-3354-4637-891c-9bcbfb0daf67-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-27ssq\" (UID: \"8f183a1c-3354-4637-891c-9bcbfb0daf67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-27ssq" Nov 03 21:03:53.366839 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:53.366689 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f183a1c-3354-4637-891c-9bcbfb0daf67-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-27ssq\" (UID: \"8f183a1c-3354-4637-891c-9bcbfb0daf67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-27ssq" Nov 03 21:03:53.366839 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:53.366735 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f183a1c-3354-4637-891c-9bcbfb0daf67-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-27ssq\" (UID: \"8f183a1c-3354-4637-891c-9bcbfb0daf67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-27ssq" Nov 03 21:03:53.366839 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:53.366785 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f183a1c-3354-4637-891c-9bcbfb0daf67-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-27ssq\" (UID: \"8f183a1c-3354-4637-891c-9bcbfb0daf67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-27ssq" Nov 03 21:03:53.367018 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:53.366935 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f183a1c-3354-4637-891c-9bcbfb0daf67-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-27ssq\" (UID: \"8f183a1c-3354-4637-891c-9bcbfb0daf67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-27ssq" Nov 03 21:03:53.367018 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:53.366966 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f183a1c-3354-4637-891c-9bcbfb0daf67-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-27ssq\" (UID: \"8f183a1c-3354-4637-891c-9bcbfb0daf67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-27ssq" Nov 03 21:03:53.367181 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:53.367148 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f183a1c-3354-4637-891c-9bcbfb0daf67-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-27ssq\" (UID: \"8f183a1c-3354-4637-891c-9bcbfb0daf67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-27ssq" Nov 03 21:03:53.367422 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:53.367397 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f183a1c-3354-4637-891c-9bcbfb0daf67-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-27ssq\" (UID: \"8f183a1c-3354-4637-891c-9bcbfb0daf67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-27ssq" Nov 03 21:03:53.367610 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:53.367517 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f183a1c-3354-4637-891c-9bcbfb0daf67-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-27ssq\" (UID: \"8f183a1c-3354-4637-891c-9bcbfb0daf67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-27ssq" Nov 03 21:03:53.367846 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:53.367817 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f183a1c-3354-4637-891c-9bcbfb0daf67-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-27ssq\" (UID: \"8f183a1c-3354-4637-891c-9bcbfb0daf67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-27ssq" Nov 03 21:03:53.372257 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:53.368577 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f183a1c-3354-4637-891c-9bcbfb0daf67-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-27ssq\" (UID: \"8f183a1c-3354-4637-891c-9bcbfb0daf67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-27ssq" Nov 03 21:03:53.374290 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:53.374259 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f183a1c-3354-4637-891c-9bcbfb0daf67-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-27ssq\" (UID: \"8f183a1c-3354-4637-891c-9bcbfb0daf67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-27ssq" Nov 03 21:03:53.376753 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:53.376726 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f183a1c-3354-4637-891c-9bcbfb0daf67-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-27ssq\" (UID: \"8f183a1c-3354-4637-891c-9bcbfb0daf67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-27ssq" Nov 03 21:03:53.431273 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:53.431240 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f183a1c-3354-4637-891c-9bcbfb0daf67-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-27ssq\" (UID: \"8f183a1c-3354-4637-891c-9bcbfb0daf67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-27ssq" Nov 03 21:03:53.431938 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:53.431916 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw8sr\" (UniqueName: \"kubernetes.io/projected/8f183a1c-3354-4637-891c-9bcbfb0daf67-kube-api-access-sw8sr\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-27ssq\" (UID: \"8f183a1c-3354-4637-891c-9bcbfb0daf67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-27ssq" Nov 03 21:03:53.548160 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:53.548070 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-27ssq" Nov 03 21:03:54.001690 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:03:54.001660 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f183a1c_3354_4637_891c_9bcbfb0daf67.slice/crio-9802010118d597136349f8ddc1a8f838963ef52d80dcadef5cc2dbbe402b5a29 WatchSource:0}: Error finding container 9802010118d597136349f8ddc1a8f838963ef52d80dcadef5cc2dbbe402b5a29: Status 404 returned error can't find the container with id 9802010118d597136349f8ddc1a8f838963ef52d80dcadef5cc2dbbe402b5a29 Nov 03 21:03:54.002408 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:54.002378 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-27ssq"] Nov 03 21:03:54.004665 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:54.004633 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:03:54.004726 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:54.004704 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:03:54.004761 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:54.004735 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:03:54.553586 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:54.550266 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-27ssq" event={"ID":"8f183a1c-3354-4637-891c-9bcbfb0daf67","Type":"ContainerStarted","Data":"134d2d0e29ddca31f2a1783a064d951bb0fb94713b9c2e4ed036e16bda28bdb2"} Nov 03 21:03:54.553586 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:54.550334 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-27ssq" event={"ID":"8f183a1c-3354-4637-891c-9bcbfb0daf67","Type":"ContainerStarted","Data":"9802010118d597136349f8ddc1a8f838963ef52d80dcadef5cc2dbbe402b5a29"} Nov 03 21:03:54.678317 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:54.678251 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-27ssq" podStartSLOduration=1.678227384 podStartE2EDuration="1.678227384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:03:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:03:54.671561793 +0000 UTC m=+1469.896881642" watchObservedRunningTime="2026-11-03 21:03:54.678227384 +0000 UTC m=+1469.903547176" Nov 03 21:03:55.548638 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:55.548605 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-27ssq" Nov 03 21:03:55.550495 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:55.550469 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-27ssq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 03 21:03:55.550642 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:55.550512 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-27ssq" podUID="8f183a1c-3354-4637-891c-9bcbfb0daf67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 03 21:03:56.548817 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:56.548779 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-27ssq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 03 21:03:56.549332 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:56.548842 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-27ssq" podUID="8f183a1c-3354-4637-891c-9bcbfb0daf67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 03 21:03:57.549605 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:57.549576 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-27ssq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 03 21:03:57.550110 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:57.550086 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-27ssq" podUID="8f183a1c-3354-4637-891c-9bcbfb0daf67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 03 21:03:58.548710 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:58.548677 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-27ssq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 03 21:03:58.548936 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:58.548752 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-27ssq" podUID="8f183a1c-3354-4637-891c-9bcbfb0daf67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 03 21:03:59.549173 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:59.549136 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-27ssq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 03 21:03:59.549698 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:03:59.549204 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-27ssq" podUID="8f183a1c-3354-4637-891c-9bcbfb0daf67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 03 21:04:00.550079 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:00.550029 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-27ssq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 03 21:04:00.550783 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:00.550749 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-27ssq" podUID="8f183a1c-3354-4637-891c-9bcbfb0daf67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 03 21:04:01.548708 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:01.548669 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-27ssq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 03 21:04:01.548880 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:01.548750 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-27ssq" podUID="8f183a1c-3354-4637-891c-9bcbfb0daf67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 03 21:04:02.607105 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:02.607072 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-27ssq" Nov 03 21:04:02.607794 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:02.607619 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-27ssq" Nov 03 21:04:02.608606 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:02.608582 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-27ssq" Nov 03 21:04:08.344236 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:08.344144 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-x954c"] Nov 03 21:04:08.351678 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:08.351643 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-x954c" Nov 03 21:04:08.393757 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:08.393724 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn"] Nov 03 21:04:08.398574 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:08.398553 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn" Nov 03 21:04:08.450599 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:08.450566 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-x954c"] Nov 03 21:04:08.455269 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:08.455236 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn\" (UID: \"fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn" Nov 03 21:04:08.455467 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:08.455287 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn\" (UID: \"fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn" Nov 03 21:04:08.455467 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:08.455393 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2195a2fd-e42f-4e90-861d-a50b75f81973-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-x954c\" (UID: \"2195a2fd-e42f-4e90-861d-a50b75f81973\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-x954c" Nov 03 21:04:08.455467 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:08.455458 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn\" (UID: \"fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn" Nov 03 21:04:08.455626 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:08.455524 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2195a2fd-e42f-4e90-861d-a50b75f81973-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-x954c\" (UID: \"2195a2fd-e42f-4e90-861d-a50b75f81973\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-x954c" Nov 03 21:04:08.455626 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:08.455596 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2195a2fd-e42f-4e90-861d-a50b75f81973-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-x954c\" (UID: \"2195a2fd-e42f-4e90-861d-a50b75f81973\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-x954c" Nov 03 21:04:08.455703 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:08.455642 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2195a2fd-e42f-4e90-861d-a50b75f81973-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-x954c\" (UID: \"2195a2fd-e42f-4e90-861d-a50b75f81973\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-x954c" Nov 03 21:04:08.455776 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:08.455760 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2195a2fd-e42f-4e90-861d-a50b75f81973-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-x954c\" (UID: \"2195a2fd-e42f-4e90-861d-a50b75f81973\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-x954c" Nov 03 21:04:08.455934 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:08.455910 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2195a2fd-e42f-4e90-861d-a50b75f81973-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-x954c\" (UID: \"2195a2fd-e42f-4e90-861d-a50b75f81973\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-x954c" Nov 03 21:04:08.455988 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:08.455973 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn\" (UID: \"fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn" Nov 03 21:04:08.456045 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:08.456026 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-474mp\" (UniqueName: \"kubernetes.io/projected/fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc-kube-api-access-474mp\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn\" (UID: \"fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn" Nov 03 21:04:08.456145 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:08.456129 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn\" (UID: \"fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn" Nov 03 21:04:08.456261 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:08.456243 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tftfp\" (UniqueName: \"kubernetes.io/projected/2195a2fd-e42f-4e90-861d-a50b75f81973-kube-api-access-tftfp\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-x954c\" (UID: \"2195a2fd-e42f-4e90-861d-a50b75f81973\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-x954c" Nov 03 21:04:08.456344 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:08.456277 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2195a2fd-e42f-4e90-861d-a50b75f81973-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-x954c\" (UID: \"2195a2fd-e42f-4e90-861d-a50b75f81973\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-x954c" Nov 03 21:04:08.456344 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:08.456310 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2195a2fd-e42f-4e90-861d-a50b75f81973-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-x954c\" (UID: \"2195a2fd-e42f-4e90-861d-a50b75f81973\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-x954c" Nov 03 21:04:08.456344 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:08.456333 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn\" (UID: \"fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn" Nov 03 21:04:08.456575 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:08.456383 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn\" (UID: \"fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn" Nov 03 21:04:08.456575 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:08.456515 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn\" (UID: \"fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn" Nov 03 21:04:08.510357 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:08.510318 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn"] Nov 03 21:04:08.558589 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:08.558539 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2195a2fd-e42f-4e90-861d-a50b75f81973-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-x954c\" (UID: \"2195a2fd-e42f-4e90-861d-a50b75f81973\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-x954c" Nov 03 21:04:08.558793 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:08.558681 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2195a2fd-e42f-4e90-861d-a50b75f81973-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-x954c\" (UID: \"2195a2fd-e42f-4e90-861d-a50b75f81973\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-x954c" Nov 03 21:04:08.558793 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:08.558738 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn\" (UID: \"fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn" Nov 03 21:04:08.558793 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:08.558763 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-474mp\" (UniqueName: \"kubernetes.io/projected/fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc-kube-api-access-474mp\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn\" (UID: \"fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn" Nov 03 21:04:08.558978 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:08.558833 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn\" (UID: \"fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn" Nov 03 21:04:08.558978 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:08.558952 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tftfp\" (UniqueName: \"kubernetes.io/projected/2195a2fd-e42f-4e90-861d-a50b75f81973-kube-api-access-tftfp\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-x954c\" (UID: \"2195a2fd-e42f-4e90-861d-a50b75f81973\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-x954c" Nov 03 21:04:08.559094 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:08.558985 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2195a2fd-e42f-4e90-861d-a50b75f81973-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-x954c\" (UID: \"2195a2fd-e42f-4e90-861d-a50b75f81973\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-x954c" Nov 03 21:04:08.559094 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:08.559023 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2195a2fd-e42f-4e90-861d-a50b75f81973-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-x954c\" (UID: \"2195a2fd-e42f-4e90-861d-a50b75f81973\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-x954c" Nov 03 21:04:08.559094 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:08.559049 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn\" (UID: \"fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn" Nov 03 21:04:08.559094 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:08.559077 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn\" (UID: \"fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn" Nov 03 21:04:08.559309 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:08.559129 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn\" (UID: \"fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn" Nov 03 21:04:08.559309 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:08.559215 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn\" (UID: \"fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn" Nov 03 21:04:08.559309 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:08.559254 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn\" (UID: \"fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn" Nov 03 21:04:08.559309 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:08.559300 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2195a2fd-e42f-4e90-861d-a50b75f81973-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-x954c\" (UID: \"2195a2fd-e42f-4e90-861d-a50b75f81973\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-x954c" Nov 03 21:04:08.559563 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:08.559339 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn\" (UID: \"fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn" Nov 03 21:04:08.559563 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:08.559369 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2195a2fd-e42f-4e90-861d-a50b75f81973-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-x954c\" (UID: \"2195a2fd-e42f-4e90-861d-a50b75f81973\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-x954c" Nov 03 21:04:08.559563 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:08.559465 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2195a2fd-e42f-4e90-861d-a50b75f81973-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-x954c\" (UID: \"2195a2fd-e42f-4e90-861d-a50b75f81973\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-x954c" Nov 03 21:04:08.559563 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:08.559503 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2195a2fd-e42f-4e90-861d-a50b75f81973-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-x954c\" (UID: \"2195a2fd-e42f-4e90-861d-a50b75f81973\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-x954c" Nov 03 21:04:08.560228 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:08.560205 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn\" (UID: \"fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn" Nov 03 21:04:08.560417 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:08.559465 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2195a2fd-e42f-4e90-861d-a50b75f81973-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-x954c\" (UID: \"2195a2fd-e42f-4e90-861d-a50b75f81973\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-x954c" Nov 03 21:04:08.560581 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:08.560558 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn\" (UID: \"fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn" Nov 03 21:04:08.560871 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:08.560848 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn\" (UID: \"fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn" Nov 03 21:04:08.560969 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:08.560902 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn\" (UID: \"fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn" Nov 03 21:04:08.561041 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:08.560988 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn\" (UID: \"fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn" Nov 03 21:04:08.561694 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:08.561482 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2195a2fd-e42f-4e90-861d-a50b75f81973-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-x954c\" (UID: \"2195a2fd-e42f-4e90-861d-a50b75f81973\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-x954c" Nov 03 21:04:08.562021 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:08.561997 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2195a2fd-e42f-4e90-861d-a50b75f81973-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-x954c\" (UID: \"2195a2fd-e42f-4e90-861d-a50b75f81973\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-x954c" Nov 03 21:04:08.562399 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:08.562378 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2195a2fd-e42f-4e90-861d-a50b75f81973-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-x954c\" (UID: \"2195a2fd-e42f-4e90-861d-a50b75f81973\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-x954c" Nov 03 21:04:08.562826 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:08.562803 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2195a2fd-e42f-4e90-861d-a50b75f81973-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-x954c\" (UID: \"2195a2fd-e42f-4e90-861d-a50b75f81973\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-x954c" Nov 03 21:04:08.562971 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:08.562818 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn\" (UID: \"fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn" Nov 03 21:04:08.563077 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:08.563058 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn\" (UID: \"fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn" Nov 03 21:04:08.563649 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:08.563626 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2195a2fd-e42f-4e90-861d-a50b75f81973-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-x954c\" (UID: \"2195a2fd-e42f-4e90-861d-a50b75f81973\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-x954c" Nov 03 21:04:08.564284 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:08.564266 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2195a2fd-e42f-4e90-861d-a50b75f81973-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-x954c\" (UID: \"2195a2fd-e42f-4e90-861d-a50b75f81973\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-x954c" Nov 03 21:04:08.620927 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:08.620846 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2195a2fd-e42f-4e90-861d-a50b75f81973-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-x954c\" (UID: \"2195a2fd-e42f-4e90-861d-a50b75f81973\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-x954c" Nov 03 21:04:08.621080 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:08.620970 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-474mp\" (UniqueName: \"kubernetes.io/projected/fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc-kube-api-access-474mp\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn\" (UID: \"fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn" Nov 03 21:04:08.621080 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:08.621011 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tftfp\" (UniqueName: \"kubernetes.io/projected/2195a2fd-e42f-4e90-861d-a50b75f81973-kube-api-access-tftfp\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-x954c\" (UID: \"2195a2fd-e42f-4e90-861d-a50b75f81973\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-x954c" Nov 03 21:04:08.621346 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:08.621322 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn\" (UID: \"fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn" Nov 03 21:04:08.670294 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:08.670261 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-x954c" Nov 03 21:04:08.714478 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:08.714423 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn" Nov 03 21:04:09.112862 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:04:09.112832 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2195a2fd_e42f_4e90_861d_a50b75f81973.slice/crio-fbb1d10d456b297a6ca33e467e8aad517bed14ef028a8132bbbe3caec5b6a31c WatchSource:0}: Error finding container fbb1d10d456b297a6ca33e467e8aad517bed14ef028a8132bbbe3caec5b6a31c: Status 404 returned error can't find the container with id fbb1d10d456b297a6ca33e467e8aad517bed14ef028a8132bbbe3caec5b6a31c Nov 03 21:04:09.115189 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:09.115112 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-x954c"] Nov 03 21:04:09.116370 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:09.116335 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:04:09.116511 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:09.116397 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:04:09.116511 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:09.116426 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:04:09.166445 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:04:09.166393 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd3eeb4b_f1a2_4ba8_b094_f22ba1fc2afc.slice/crio-6bd5edf0e0cf9bf7ecd638e4ec2a1ffc6ed5f24f58e6bcfe1e2df584b8f028c3 WatchSource:0}: Error finding container 6bd5edf0e0cf9bf7ecd638e4ec2a1ffc6ed5f24f58e6bcfe1e2df584b8f028c3: Status 404 returned error can't find the container with id 6bd5edf0e0cf9bf7ecd638e4ec2a1ffc6ed5f24f58e6bcfe1e2df584b8f028c3 Nov 03 21:04:09.167663 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:09.167637 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn"] Nov 03 21:04:09.169829 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:09.169796 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:04:09.169945 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:09.169879 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:04:09.169945 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:09.169934 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:04:09.726222 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:09.726185 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-x954c" event={"ID":"2195a2fd-e42f-4e90-861d-a50b75f81973","Type":"ContainerStarted","Data":"38753558348ff64f5e879a4041a51532b05251bd1d16f6c0200cca6e992cd8ef"} Nov 03 21:04:09.726895 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:09.726851 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-x954c" event={"ID":"2195a2fd-e42f-4e90-861d-a50b75f81973","Type":"ContainerStarted","Data":"fbb1d10d456b297a6ca33e467e8aad517bed14ef028a8132bbbe3caec5b6a31c"} Nov 03 21:04:09.729246 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:09.729216 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn" event={"ID":"fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc","Type":"ContainerStarted","Data":"32dca95a4802050f05da86005a6ea74031dc7b0017bbe0276b03e076fefd6383"} Nov 03 21:04:09.729381 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:09.729272 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn" event={"ID":"fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc","Type":"ContainerStarted","Data":"6bd5edf0e0cf9bf7ecd638e4ec2a1ffc6ed5f24f58e6bcfe1e2df584b8f028c3"} Nov 03 21:04:09.862114 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:09.862040 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-x954c" podStartSLOduration=1.862018698 podStartE2EDuration="1.862018698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:04:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:04:09.850933444 +0000 UTC m=+1485.076253275" watchObservedRunningTime="2026-11-03 21:04:09.862018698 +0000 UTC m=+1485.087338497" Nov 03 21:04:10.523461 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:10.520351 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn" podStartSLOduration=2.520329151 podStartE2EDuration="2.520329151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:04:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:04:10.003691521 +0000 UTC m=+1485.229011313" watchObservedRunningTime="2026-11-03 21:04:10.520329151 +0000 UTC m=+1485.745648966" Nov 03 21:04:10.561574 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:10.561537 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:04:10.673461 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:10.672982 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-x954c" Nov 03 21:04:10.680659 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:10.678788 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-x954c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 03 21:04:10.680659 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:10.678847 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-x954c" podUID="2195a2fd-e42f-4e90-861d-a50b75f81973" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 03 21:04:10.733772 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:10.727601 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn" Nov 03 21:04:10.733772 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:10.727773 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Nov 03 21:04:10.733772 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:10.727829 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn" podUID="fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Nov 03 21:04:11.678457 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:11.671923 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-x954c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 03 21:04:11.678457 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:11.672029 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-x954c" podUID="2195a2fd-e42f-4e90-861d-a50b75f81973" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 03 21:04:11.715469 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:11.714999 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Nov 03 21:04:11.715469 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:11.715060 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn" podUID="fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Nov 03 21:04:12.705738 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:12.705700 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-x954c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 03 21:04:12.706342 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:12.706307 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-x954c" podUID="2195a2fd-e42f-4e90-861d-a50b75f81973" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 03 21:04:12.743210 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:12.742202 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Nov 03 21:04:12.743210 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:12.742268 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn" podUID="fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Nov 03 21:04:13.693471 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:13.692747 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-x954c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 03 21:04:13.693471 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:13.692814 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-x954c" podUID="2195a2fd-e42f-4e90-861d-a50b75f81973" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 03 21:04:13.737515 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:13.730465 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Nov 03 21:04:13.737515 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:13.730529 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn" podUID="fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Nov 03 21:04:14.716652 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:14.716615 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Nov 03 21:04:14.717008 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:14.716981 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn" podUID="fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Nov 03 21:04:14.717615 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:14.717573 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-x954c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 03 21:04:14.717786 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:14.717768 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-x954c" podUID="2195a2fd-e42f-4e90-861d-a50b75f81973" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 03 21:04:15.675925 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:15.675876 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-x954c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 03 21:04:15.676590 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:15.676556 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-x954c" podUID="2195a2fd-e42f-4e90-861d-a50b75f81973" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 03 21:04:15.723460 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:15.715881 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Nov 03 21:04:15.723460 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:15.715955 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn" podUID="fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Nov 03 21:04:16.408645 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:04:16.408583 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:04:16.672525 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:16.671933 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-x954c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 03 21:04:16.672525 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:16.672005 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-x954c" podUID="2195a2fd-e42f-4e90-861d-a50b75f81973" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 03 21:04:16.715396 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:16.715356 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Nov 03 21:04:16.715847 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:16.715452 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn" podUID="fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Nov 03 21:04:17.671962 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:17.671916 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-x954c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 03 21:04:17.672294 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:17.672262 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-x954c" podUID="2195a2fd-e42f-4e90-861d-a50b75f81973" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 03 21:04:17.715538 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:17.715498 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Nov 03 21:04:17.715965 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:17.715578 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn" podUID="fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Nov 03 21:04:19.711005 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:19.710968 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn" Nov 03 21:04:19.711574 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:19.711558 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn" Nov 03 21:04:19.711979 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:19.711945 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-x954c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:04:19.712136 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:19.712114 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-x954c" podUID="2195a2fd-e42f-4e90-861d-a50b75f81973" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:04:19.712561 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:19.712540 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn" Nov 03 21:04:19.810094 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:19.810059 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-x954c" Nov 03 21:04:19.810508 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:19.810476 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-x954c" Nov 03 21:04:19.811580 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:19.811562 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-x954c" Nov 03 21:04:23.130928 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:23.130840 4305 generic.go:356] "Generic (PLEG): container finished" podID="8ecb5391-cd65-4987-8f36-95ac0f5371dd" containerID="53264c14ee737b1b67b6bf8dea0f1dda0bdc2fd15096e018205a2ec05dc024b4" exitCode=0 Nov 03 21:04:23.130928 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:23.130914 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4" event={"ID":"8ecb5391-cd65-4987-8f36-95ac0f5371dd","Type":"ContainerDied","Data":"53264c14ee737b1b67b6bf8dea0f1dda0bdc2fd15096e018205a2ec05dc024b4"} Nov 03 21:04:23.131492 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:23.131414 4305 scope.go:122] "RemoveContainer" containerID="53264c14ee737b1b67b6bf8dea0f1dda0bdc2fd15096e018205a2ec05dc024b4" Nov 03 21:04:23.133358 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:23.133331 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:04:23.133423 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:23.133410 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:04:23.133513 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:23.133462 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:04:24.142999 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:24.142955 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4" event={"ID":"8ecb5391-cd65-4987-8f36-95ac0f5371dd","Type":"ContainerStarted","Data":"8ae5bb6467106d4eb782f7a101be0dd69801771d59e22e8eb477027dadf97e20"} Nov 03 21:04:24.847465 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:24.847409 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4" Nov 03 21:04:24.847691 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:24.847623 4305 patch_prober.go:29] interesting pod/e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 03 21:04:24.847691 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:24.847668 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4" podUID="8ecb5391-cd65-4987-8f36-95ac0f5371dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 03 21:04:25.847793 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:25.847755 4305 patch_prober.go:29] interesting pod/e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 03 21:04:25.848251 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:25.847824 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4" podUID="8ecb5391-cd65-4987-8f36-95ac0f5371dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 03 21:04:26.847728 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:26.847684 4305 patch_prober.go:29] interesting pod/e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 03 21:04:26.847935 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:26.847749 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4" podUID="8ecb5391-cd65-4987-8f36-95ac0f5371dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 03 21:04:27.847927 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:27.847878 4305 patch_prober.go:29] interesting pod/e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 03 21:04:27.848484 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:27.847956 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4" podUID="8ecb5391-cd65-4987-8f36-95ac0f5371dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 03 21:04:28.847739 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:28.847704 4305 patch_prober.go:29] interesting pod/e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 03 21:04:28.847950 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:28.847776 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4" podUID="8ecb5391-cd65-4987-8f36-95ac0f5371dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 03 21:04:28.916194 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:28.916157 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c704c5a1-rtzf2"] Nov 03 21:04:28.921548 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:28.921523 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c704c5a1-rtzf2" Nov 03 21:04:28.983460 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:28.975870 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:04:28.983460 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:28.976015 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77vtd\"" Nov 03 21:04:28.983460 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:28.982227 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c704c5a1-rtzf2"] Nov 03 21:04:29.017931 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:29.017886 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2084750-79dc-4df9-bb37-7a1fff802b05-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c704c5a1-rtzf2\" (UID: \"b2084750-79dc-4df9-bb37-7a1fff802b05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c704c5a1-rtzf2" Nov 03 21:04:29.018184 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:29.018138 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp6l2\" (UniqueName: \"kubernetes.io/projected/b2084750-79dc-4df9-bb37-7a1fff802b05-kube-api-access-qp6l2\") pod \"maas-api-revoke-keys-e2e-ait-c704c5a1-rtzf2\" (UID: \"b2084750-79dc-4df9-bb37-7a1fff802b05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c704c5a1-rtzf2" Nov 03 21:04:29.018369 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:29.018345 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2084750-79dc-4df9-bb37-7a1fff802b05-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c704c5a1-rtzf2\" (UID: \"b2084750-79dc-4df9-bb37-7a1fff802b05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c704c5a1-rtzf2" Nov 03 21:04:29.021693 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:29.021646 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:04:29.021693 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:29.021646 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:04:29.120498 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:29.120354 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp6l2\" (UniqueName: \"kubernetes.io/projected/b2084750-79dc-4df9-bb37-7a1fff802b05-kube-api-access-qp6l2\") pod \"maas-api-revoke-keys-e2e-ait-c704c5a1-rtzf2\" (UID: \"b2084750-79dc-4df9-bb37-7a1fff802b05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c704c5a1-rtzf2" Nov 03 21:04:29.120702 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:29.120506 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2084750-79dc-4df9-bb37-7a1fff802b05-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c704c5a1-rtzf2\" (UID: \"b2084750-79dc-4df9-bb37-7a1fff802b05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c704c5a1-rtzf2" Nov 03 21:04:29.120702 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:29.120575 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2084750-79dc-4df9-bb37-7a1fff802b05-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c704c5a1-rtzf2\" (UID: \"b2084750-79dc-4df9-bb37-7a1fff802b05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c704c5a1-rtzf2" Nov 03 21:04:29.121308 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:29.121285 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2084750-79dc-4df9-bb37-7a1fff802b05-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c704c5a1-rtzf2\" (UID: \"b2084750-79dc-4df9-bb37-7a1fff802b05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c704c5a1-rtzf2" Nov 03 21:04:29.123247 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:29.123216 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2084750-79dc-4df9-bb37-7a1fff802b05-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c704c5a1-rtzf2\" (UID: \"b2084750-79dc-4df9-bb37-7a1fff802b05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c704c5a1-rtzf2" Nov 03 21:04:29.182102 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:29.182067 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp6l2\" (UniqueName: \"kubernetes.io/projected/b2084750-79dc-4df9-bb37-7a1fff802b05-kube-api-access-qp6l2\") pod \"maas-api-revoke-keys-e2e-ait-c704c5a1-rtzf2\" (UID: \"b2084750-79dc-4df9-bb37-7a1fff802b05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c704c5a1-rtzf2" Nov 03 21:04:29.237546 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:29.237425 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c704c5a1-rtzf2" Nov 03 21:04:29.660898 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:04:29.660863 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2084750_79dc_4df9_bb37_7a1fff802b05.slice/crio-80e2e30f064289dd235aed09b51cc96ca98fafda15fb6dce0f88b2ba9b1b6cfd WatchSource:0}: Error finding container 80e2e30f064289dd235aed09b51cc96ca98fafda15fb6dce0f88b2ba9b1b6cfd: Status 404 returned error can't find the container with id 80e2e30f064289dd235aed09b51cc96ca98fafda15fb6dce0f88b2ba9b1b6cfd Nov 03 21:04:29.662382 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:29.662361 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c704c5a1-rtzf2"] Nov 03 21:04:29.853578 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:29.853549 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4" Nov 03 21:04:29.854104 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:29.854079 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4" Nov 03 21:04:29.855273 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:29.855255 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4" Nov 03 21:04:30.237483 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:30.237424 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c704c5a1-rtzf2" event={"ID":"b2084750-79dc-4df9-bb37-7a1fff802b05","Type":"ContainerStarted","Data":"d788313d5dd90b3cb4e77528fe9d818d43ec2ac3c03196a26e45f9cb6b92a347"} Nov 03 21:04:30.237683 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:30.237509 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c704c5a1-rtzf2" event={"ID":"b2084750-79dc-4df9-bb37-7a1fff802b05","Type":"ContainerStarted","Data":"80e2e30f064289dd235aed09b51cc96ca98fafda15fb6dce0f88b2ba9b1b6cfd"} Nov 03 21:04:30.357852 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:30.357742 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c704c5a1-rtzf2" podStartSLOduration=2.357727824 podStartE2EDuration="2.357727824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:04:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:04:30.353942778 +0000 UTC m=+1505.579262572" watchObservedRunningTime="2026-11-03 21:04:30.357727824 +0000 UTC m=+1505.583047618" Nov 03 21:04:31.251035 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:31.250990 4305 generic.go:356] "Generic (PLEG): container finished" podID="b2084750-79dc-4df9-bb37-7a1fff802b05" containerID="d788313d5dd90b3cb4e77528fe9d818d43ec2ac3c03196a26e45f9cb6b92a347" exitCode=0 Nov 03 21:04:31.251539 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:31.251071 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c704c5a1-rtzf2" event={"ID":"b2084750-79dc-4df9-bb37-7a1fff802b05","Type":"ContainerDied","Data":"d788313d5dd90b3cb4e77528fe9d818d43ec2ac3c03196a26e45f9cb6b92a347"} Nov 03 21:04:32.448386 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:32.448357 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c704c5a1-rtzf2" Nov 03 21:04:32.579799 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:32.579417 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2084750-79dc-4df9-bb37-7a1fff802b05-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2084750-79dc-4df9-bb37-7a1fff802b05-maas-api-openshift-ca\") pod \"b2084750-79dc-4df9-bb37-7a1fff802b05\" (UID: \"b2084750-79dc-4df9-bb37-7a1fff802b05\") " Nov 03 21:04:32.579799 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:32.579748 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2084750-79dc-4df9-bb37-7a1fff802b05-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2084750-79dc-4df9-bb37-7a1fff802b05-maas-api-xks-ca\") pod \"b2084750-79dc-4df9-bb37-7a1fff802b05\" (UID: \"b2084750-79dc-4df9-bb37-7a1fff802b05\") " Nov 03 21:04:32.580102 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:32.579853 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2084750-79dc-4df9-bb37-7a1fff802b05-kube-api-access-qp6l2\" (UniqueName: \"kubernetes.io/projected/b2084750-79dc-4df9-bb37-7a1fff802b05-kube-api-access-qp6l2\") pod \"b2084750-79dc-4df9-bb37-7a1fff802b05\" (UID: \"b2084750-79dc-4df9-bb37-7a1fff802b05\") " Nov 03 21:04:32.580458 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:32.580407 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2084750-79dc-4df9-bb37-7a1fff802b05-maas-api-openshift-ca" pod "b2084750-79dc-4df9-bb37-7a1fff802b05" (UID: "b2084750-79dc-4df9-bb37-7a1fff802b05"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:04:32.580982 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:32.580954 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2084750-79dc-4df9-bb37-7a1fff802b05-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:04:32.588716 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:32.588675 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2084750-79dc-4df9-bb37-7a1fff802b05-kube-api-access-qp6l2" pod "b2084750-79dc-4df9-bb37-7a1fff802b05" (UID: "b2084750-79dc-4df9-bb37-7a1fff802b05"). InnerVolumeSpecName "kube-api-access-qp6l2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:04:32.590046 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:32.589993 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2084750-79dc-4df9-bb37-7a1fff802b05-maas-api-xks-ca" pod "b2084750-79dc-4df9-bb37-7a1fff802b05" (UID: "b2084750-79dc-4df9-bb37-7a1fff802b05"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:04:32.683202 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:32.683163 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp6l2\" (UniqueName: \"kubernetes.io/projected/b2084750-79dc-4df9-bb37-7a1fff802b05-kube-api-access-qp6l2\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:04:32.683202 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:32.683198 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2084750-79dc-4df9-bb37-7a1fff802b05-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:04:33.271336 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:33.271250 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c704c5a1-rtzf2" Nov 03 21:04:33.271551 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:33.271251 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c704c5a1-rtzf2" event={"ID":"b2084750-79dc-4df9-bb37-7a1fff802b05","Type":"ContainerDied","Data":"80e2e30f064289dd235aed09b51cc96ca98fafda15fb6dce0f88b2ba9b1b6cfd"} Nov 03 21:04:33.271551 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:33.271375 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80e2e30f064289dd235aed09b51cc96ca98fafda15fb6dce0f88b2ba9b1b6cfd" Nov 03 21:04:35.589607 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:35.589569 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-46590df2-tzq6b"] Nov 03 21:04:35.590663 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:35.590642 4305 cpu_manager.go:414] "containerMap: removing container" podUID="b2084750-79dc-4df9-bb37-7a1fff802b05" containerName="revoke-keys" Nov 03 21:04:35.590725 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:35.590667 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2084750-79dc-4df9-bb37-7a1fff802b05" containerName="revoke-keys" Nov 03 21:04:35.590899 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:35.590887 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2084750-79dc-4df9-bb37-7a1fff802b05" containerName="revoke-keys" Nov 03 21:04:35.602281 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:35.602254 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-46590df2-tzq6b" Nov 03 21:04:35.653242 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:35.653206 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-46590df2-tzq6b"] Nov 03 21:04:35.695617 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:35.695579 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:04:35.695617 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:35.695590 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:04:35.695870 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:35.695714 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77vtd\"" Nov 03 21:04:35.704353 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:35.704327 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:04:35.734222 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:35.734185 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00169062-4b97-41c7-afa8-2a3e6f6678a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-46590df2-tzq6b\" (UID: \"00169062-4b97-41c7-afa8-2a3e6f6678a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-46590df2-tzq6b" Nov 03 21:04:35.734417 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:35.734339 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00169062-4b97-41c7-afa8-2a3e6f6678a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-46590df2-tzq6b\" (UID: \"00169062-4b97-41c7-afa8-2a3e6f6678a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-46590df2-tzq6b" Nov 03 21:04:35.734623 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:35.734600 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25jd6\" (UniqueName: \"kubernetes.io/projected/00169062-4b97-41c7-afa8-2a3e6f6678a3-kube-api-access-25jd6\") pod \"maas-api-revoke-keys-e2e-migrate-46590df2-tzq6b\" (UID: \"00169062-4b97-41c7-afa8-2a3e6f6678a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-46590df2-tzq6b" Nov 03 21:04:35.836093 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:35.836050 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25jd6\" (UniqueName: \"kubernetes.io/projected/00169062-4b97-41c7-afa8-2a3e6f6678a3-kube-api-access-25jd6\") pod \"maas-api-revoke-keys-e2e-migrate-46590df2-tzq6b\" (UID: \"00169062-4b97-41c7-afa8-2a3e6f6678a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-46590df2-tzq6b" Nov 03 21:04:35.836290 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:35.836221 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00169062-4b97-41c7-afa8-2a3e6f6678a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-46590df2-tzq6b\" (UID: \"00169062-4b97-41c7-afa8-2a3e6f6678a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-46590df2-tzq6b" Nov 03 21:04:35.836385 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:35.836303 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00169062-4b97-41c7-afa8-2a3e6f6678a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-46590df2-tzq6b\" (UID: \"00169062-4b97-41c7-afa8-2a3e6f6678a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-46590df2-tzq6b" Nov 03 21:04:35.837050 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:35.837028 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00169062-4b97-41c7-afa8-2a3e6f6678a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-46590df2-tzq6b\" (UID: \"00169062-4b97-41c7-afa8-2a3e6f6678a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-46590df2-tzq6b" Nov 03 21:04:35.839133 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:35.839107 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00169062-4b97-41c7-afa8-2a3e6f6678a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-46590df2-tzq6b\" (UID: \"00169062-4b97-41c7-afa8-2a3e6f6678a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-46590df2-tzq6b" Nov 03 21:04:35.897678 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:35.897650 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25jd6\" (UniqueName: \"kubernetes.io/projected/00169062-4b97-41c7-afa8-2a3e6f6678a3-kube-api-access-25jd6\") pod \"maas-api-revoke-keys-e2e-migrate-46590df2-tzq6b\" (UID: \"00169062-4b97-41c7-afa8-2a3e6f6678a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-46590df2-tzq6b" Nov 03 21:04:35.917905 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:35.917872 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-46590df2-tzq6b" Nov 03 21:04:36.309295 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:04:36.309252 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00169062_4b97_41c7_afa8_2a3e6f6678a3.slice/crio-fff66bf15d188b7da0332e3105d60a4fc9ffc94fe7c0c787c003330b8d62f9cd WatchSource:0}: Error finding container fff66bf15d188b7da0332e3105d60a4fc9ffc94fe7c0c787c003330b8d62f9cd: Status 404 returned error can't find the container with id fff66bf15d188b7da0332e3105d60a4fc9ffc94fe7c0c787c003330b8d62f9cd Nov 03 21:04:36.311339 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:36.311317 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-46590df2-tzq6b"] Nov 03 21:04:37.311520 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:37.311483 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-46590df2-tzq6b" event={"ID":"00169062-4b97-41c7-afa8-2a3e6f6678a3","Type":"ContainerStarted","Data":"9d1a3c0e8fa874ede2e654b37c318cc124a719cc769bb43040b87736e8603472"} Nov 03 21:04:37.312052 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:37.311551 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-46590df2-tzq6b" event={"ID":"00169062-4b97-41c7-afa8-2a3e6f6678a3","Type":"ContainerStarted","Data":"fff66bf15d188b7da0332e3105d60a4fc9ffc94fe7c0c787c003330b8d62f9cd"} Nov 03 21:04:37.430581 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:37.430525 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-46590df2-tzq6b" podStartSLOduration=2.430508736 podStartE2EDuration="2.430508736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:04:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:04:37.426906448 +0000 UTC m=+1512.652226258" watchObservedRunningTime="2026-11-03 21:04:37.430508736 +0000 UTC m=+1512.655828526" Nov 03 21:04:38.322568 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:38.322528 4305 generic.go:356] "Generic (PLEG): container finished" podID="00169062-4b97-41c7-afa8-2a3e6f6678a3" containerID="9d1a3c0e8fa874ede2e654b37c318cc124a719cc769bb43040b87736e8603472" exitCode=0 Nov 03 21:04:38.323064 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:38.322606 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-46590df2-tzq6b" event={"ID":"00169062-4b97-41c7-afa8-2a3e6f6678a3","Type":"ContainerDied","Data":"9d1a3c0e8fa874ede2e654b37c318cc124a719cc769bb43040b87736e8603472"} Nov 03 21:04:39.523119 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:39.523094 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-46590df2-tzq6b" Nov 03 21:04:39.597329 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:39.597243 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00169062-4b97-41c7-afa8-2a3e6f6678a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00169062-4b97-41c7-afa8-2a3e6f6678a3-maas-api-xks-ca\") pod \"00169062-4b97-41c7-afa8-2a3e6f6678a3\" (UID: \"00169062-4b97-41c7-afa8-2a3e6f6678a3\") " Nov 03 21:04:39.597512 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:39.597448 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00169062-4b97-41c7-afa8-2a3e6f6678a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00169062-4b97-41c7-afa8-2a3e6f6678a3-maas-api-openshift-ca\") pod \"00169062-4b97-41c7-afa8-2a3e6f6678a3\" (UID: \"00169062-4b97-41c7-afa8-2a3e6f6678a3\") " Nov 03 21:04:39.597512 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:39.597502 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00169062-4b97-41c7-afa8-2a3e6f6678a3-kube-api-access-25jd6\" (UniqueName: \"kubernetes.io/projected/00169062-4b97-41c7-afa8-2a3e6f6678a3-kube-api-access-25jd6\") pod \"00169062-4b97-41c7-afa8-2a3e6f6678a3\" (UID: \"00169062-4b97-41c7-afa8-2a3e6f6678a3\") " Nov 03 21:04:39.597875 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:39.597845 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00169062-4b97-41c7-afa8-2a3e6f6678a3-maas-api-openshift-ca" pod "00169062-4b97-41c7-afa8-2a3e6f6678a3" (UID: "00169062-4b97-41c7-afa8-2a3e6f6678a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:04:39.598344 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:39.598322 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00169062-4b97-41c7-afa8-2a3e6f6678a3-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:04:39.599632 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:39.599604 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00169062-4b97-41c7-afa8-2a3e6f6678a3-maas-api-xks-ca" pod "00169062-4b97-41c7-afa8-2a3e6f6678a3" (UID: "00169062-4b97-41c7-afa8-2a3e6f6678a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:04:39.599787 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:39.599768 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00169062-4b97-41c7-afa8-2a3e6f6678a3-kube-api-access-25jd6" pod "00169062-4b97-41c7-afa8-2a3e6f6678a3" (UID: "00169062-4b97-41c7-afa8-2a3e6f6678a3"). InnerVolumeSpecName "kube-api-access-25jd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:04:39.700402 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:39.700358 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00169062-4b97-41c7-afa8-2a3e6f6678a3-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:04:39.700402 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:39.700403 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25jd6\" (UniqueName: \"kubernetes.io/projected/00169062-4b97-41c7-afa8-2a3e6f6678a3-kube-api-access-25jd6\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:04:40.342675 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:40.342643 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-46590df2-tzq6b" Nov 03 21:04:40.342675 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:40.342662 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-46590df2-tzq6b" event={"ID":"00169062-4b97-41c7-afa8-2a3e6f6678a3","Type":"ContainerDied","Data":"fff66bf15d188b7da0332e3105d60a4fc9ffc94fe7c0c787c003330b8d62f9cd"} Nov 03 21:04:40.342912 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:40.342708 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fff66bf15d188b7da0332e3105d60a4fc9ffc94fe7c0c787c003330b8d62f9cd" Nov 03 21:04:42.072965 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:42.072924 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ed16b66-kdsks"] Nov 03 21:04:42.105034 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:42.104989 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ed16b66-kdsks"] Nov 03 21:04:42.897909 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:42.897873 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57fea02a-046f-4cd2-9373-09d1c47de7d1" path="/var/lib/kubelet/pods/57fea02a-046f-4cd2-9373-09d1c47de7d1/volumes" Nov 03 21:04:46.327162 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:04:46.327126 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:04:47.143248 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:47.143200 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-46590df2-7d46fd668-6cc2h"] Nov 03 21:04:47.283558 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:47.283525 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-46590df2-66f54d6b9f-b48rg"] Nov 03 21:04:52.143554 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:52.143499 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-46590df2-7d46fd668-6cc2h" podUID="3320ca8a-f6ac-44c1-a877-a62db11a904c" containerName="payload-processing" containerID="cri-o://9aa6207e126cab6ebd5198196fee86190fa28c39be32f9335bf3fcbd4246c5b8" gracePeriod=25 Nov 03 21:04:52.284240 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:52.284202 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-46590df2-66f54d6b9f-b48rg" podUID="47183bab-92a7-49cc-a7ef-446106ac055d" containerName="payload-pre-processing" containerID="cri-o://5c51283e79e4efe9e1f01bd9a15a700d3d4dace319264c013304ffba19666ca9" gracePeriod=25 Nov 03 21:04:52.465848 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:52.465794 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-46590df2-7d46fd668-6cc2h" Nov 03 21:04:52.530259 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:52.530151 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3320ca8a-f6ac-44c1-a877-a62db11a904c-kube-api-access-9mqf7\" (UniqueName: \"kubernetes.io/projected/3320ca8a-f6ac-44c1-a877-a62db11a904c-kube-api-access-9mqf7\") pod \"3320ca8a-f6ac-44c1-a877-a62db11a904c\" (UID: \"3320ca8a-f6ac-44c1-a877-a62db11a904c\") " Nov 03 21:04:52.530479 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:52.530418 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3320ca8a-f6ac-44c1-a877-a62db11a904c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3320ca8a-f6ac-44c1-a877-a62db11a904c-plugins-config-volume\") pod \"3320ca8a-f6ac-44c1-a877-a62db11a904c\" (UID: \"3320ca8a-f6ac-44c1-a877-a62db11a904c\") " Nov 03 21:04:52.537804 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:52.537704 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3320ca8a-f6ac-44c1-a877-a62db11a904c-plugins-config-volume" pod "3320ca8a-f6ac-44c1-a877-a62db11a904c" (UID: "3320ca8a-f6ac-44c1-a877-a62db11a904c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:04:52.542518 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:52.542482 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3320ca8a-f6ac-44c1-a877-a62db11a904c-kube-api-access-9mqf7" pod "3320ca8a-f6ac-44c1-a877-a62db11a904c" (UID: "3320ca8a-f6ac-44c1-a877-a62db11a904c"). InnerVolumeSpecName "kube-api-access-9mqf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:04:52.559343 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:52.559308 4305 generic.go:356] "Generic (PLEG): container finished" podID="3320ca8a-f6ac-44c1-a877-a62db11a904c" containerID="9aa6207e126cab6ebd5198196fee86190fa28c39be32f9335bf3fcbd4246c5b8" exitCode=0 Nov 03 21:04:52.559580 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:52.559522 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-46590df2-7d46fd668-6cc2h" event={"ID":"3320ca8a-f6ac-44c1-a877-a62db11a904c","Type":"ContainerDied","Data":"9aa6207e126cab6ebd5198196fee86190fa28c39be32f9335bf3fcbd4246c5b8"} Nov 03 21:04:52.559670 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:52.559580 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-46590df2-7d46fd668-6cc2h" event={"ID":"3320ca8a-f6ac-44c1-a877-a62db11a904c","Type":"ContainerDied","Data":"41f3e949d5314731d4cbfe47029bf1a30db3404f772f6b6cbafc725dccfe4690"} Nov 03 21:04:52.559670 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:52.559611 4305 scope.go:122] "RemoveContainer" containerID="9aa6207e126cab6ebd5198196fee86190fa28c39be32f9335bf3fcbd4246c5b8" Nov 03 21:04:52.559822 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:52.559796 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-46590df2-7d46fd668-6cc2h" Nov 03 21:04:52.568168 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:52.568131 4305 generic.go:356] "Generic (PLEG): container finished" podID="47183bab-92a7-49cc-a7ef-446106ac055d" containerID="5c51283e79e4efe9e1f01bd9a15a700d3d4dace319264c013304ffba19666ca9" exitCode=0 Nov 03 21:04:52.568322 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:52.568221 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-46590df2-66f54d6b9f-b48rg" event={"ID":"47183bab-92a7-49cc-a7ef-446106ac055d","Type":"ContainerDied","Data":"5c51283e79e4efe9e1f01bd9a15a700d3d4dace319264c013304ffba19666ca9"} Nov 03 21:04:52.587560 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:52.587470 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-46590df2-66f54d6b9f-b48rg" Nov 03 21:04:52.592423 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:52.592320 4305 scope.go:122] "RemoveContainer" containerID="9aa6207e126cab6ebd5198196fee86190fa28c39be32f9335bf3fcbd4246c5b8" Nov 03 21:04:52.594620 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:04:52.594293 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9aa6207e126cab6ebd5198196fee86190fa28c39be32f9335bf3fcbd4246c5b8\": container with ID starting with 9aa6207e126cab6ebd5198196fee86190fa28c39be32f9335bf3fcbd4246c5b8 not found: ID does not exist" containerID="9aa6207e126cab6ebd5198196fee86190fa28c39be32f9335bf3fcbd4246c5b8" Nov 03 21:04:52.594620 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:52.594335 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9aa6207e126cab6ebd5198196fee86190fa28c39be32f9335bf3fcbd4246c5b8"} err="failed to get container status \"9aa6207e126cab6ebd5198196fee86190fa28c39be32f9335bf3fcbd4246c5b8\": rpc error: code = NotFound desc = could not find container \"9aa6207e126cab6ebd5198196fee86190fa28c39be32f9335bf3fcbd4246c5b8\": container with ID starting with 9aa6207e126cab6ebd5198196fee86190fa28c39be32f9335bf3fcbd4246c5b8 not found: ID does not exist" Nov 03 21:04:52.632073 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:52.632022 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47183bab-92a7-49cc-a7ef-446106ac055d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47183bab-92a7-49cc-a7ef-446106ac055d-plugins-config-volume\") pod \"47183bab-92a7-49cc-a7ef-446106ac055d\" (UID: \"47183bab-92a7-49cc-a7ef-446106ac055d\") " Nov 03 21:04:52.632591 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:52.632352 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47183bab-92a7-49cc-a7ef-446106ac055d-kube-api-access-495xj\" (UniqueName: \"kubernetes.io/projected/47183bab-92a7-49cc-a7ef-446106ac055d-kube-api-access-495xj\") pod \"47183bab-92a7-49cc-a7ef-446106ac055d\" (UID: \"47183bab-92a7-49cc-a7ef-446106ac055d\") " Nov 03 21:04:52.633316 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:52.632983 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47183bab-92a7-49cc-a7ef-446106ac055d-plugins-config-volume" pod "47183bab-92a7-49cc-a7ef-446106ac055d" (UID: "47183bab-92a7-49cc-a7ef-446106ac055d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:04:52.633451 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:52.633358 4305 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47183bab-92a7-49cc-a7ef-446106ac055d-plugins-config-volume\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:04:52.633451 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:52.633382 4305 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3320ca8a-f6ac-44c1-a877-a62db11a904c-plugins-config-volume\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:04:52.633451 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:52.633398 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mqf7\" (UniqueName: \"kubernetes.io/projected/3320ca8a-f6ac-44c1-a877-a62db11a904c-kube-api-access-9mqf7\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:04:52.636009 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:52.635977 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47183bab-92a7-49cc-a7ef-446106ac055d-kube-api-access-495xj" pod "47183bab-92a7-49cc-a7ef-446106ac055d" (UID: "47183bab-92a7-49cc-a7ef-446106ac055d"). InnerVolumeSpecName "kube-api-access-495xj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:04:52.705693 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:52.705603 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-46590df2-7d46fd668-6cc2h"] Nov 03 21:04:52.714197 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:52.714161 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-46590df2-7d46fd668-6cc2h"] Nov 03 21:04:52.734887 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:52.734855 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-495xj\" (UniqueName: \"kubernetes.io/projected/47183bab-92a7-49cc-a7ef-446106ac055d-kube-api-access-495xj\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:04:52.903150 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:52.903107 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3320ca8a-f6ac-44c1-a877-a62db11a904c" path="/var/lib/kubelet/pods/3320ca8a-f6ac-44c1-a877-a62db11a904c/volumes" Nov 03 21:04:53.581303 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:53.581224 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-46590df2-66f54d6b9f-b48rg" Nov 03 21:04:53.581303 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:53.581244 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-46590df2-66f54d6b9f-b48rg" event={"ID":"47183bab-92a7-49cc-a7ef-446106ac055d","Type":"ContainerDied","Data":"3c8fc1452504380e2834582260ee2e00cd49af7acc0f4df5f7b20c8a3fd030f4"} Nov 03 21:04:53.581848 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:53.581307 4305 scope.go:122] "RemoveContainer" containerID="5c51283e79e4efe9e1f01bd9a15a700d3d4dace319264c013304ffba19666ca9" Nov 03 21:04:53.752243 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:53.752207 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-46590df2-66f54d6b9f-b48rg"] Nov 03 21:04:53.801255 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:53.801220 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-46590df2-66f54d6b9f-b48rg"] Nov 03 21:04:54.037759 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:54.036527 4305 patch_prober.go:29] interesting pod/e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": read tcp 10.134.0.2:58950->10.134.0.145:15021: read: connection reset by peer" start-of-body= Nov 03 21:04:54.037759 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:54.036643 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz" podUID="95dd3f17-e145-49ec-8fe2-6148c3f07c80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": read tcp 10.134.0.2:58950->10.134.0.145:15021: read: connection reset by peer" Nov 03 21:04:54.595015 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:54.594983 4305 generic.go:356] "Generic (PLEG): container finished" podID="95dd3f17-e145-49ec-8fe2-6148c3f07c80" containerID="18c6ebd03bc3927d7424bc82cfe496702ce416880bdeeaa6523167f626cbe951" exitCode=0 Nov 03 21:04:54.595536 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:54.595056 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz" event={"ID":"95dd3f17-e145-49ec-8fe2-6148c3f07c80","Type":"ContainerDied","Data":"18c6ebd03bc3927d7424bc82cfe496702ce416880bdeeaa6523167f626cbe951"} Nov 03 21:04:54.595665 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:54.595645 4305 scope.go:122] "RemoveContainer" containerID="18c6ebd03bc3927d7424bc82cfe496702ce416880bdeeaa6523167f626cbe951" Nov 03 21:04:54.597498 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:54.597466 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:04:54.597625 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:54.597583 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:04:54.597687 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:54.597636 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:04:54.900804 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:54.900764 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47183bab-92a7-49cc-a7ef-446106ac055d" path="/var/lib/kubelet/pods/47183bab-92a7-49cc-a7ef-446106ac055d/volumes" Nov 03 21:04:55.607527 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:55.607492 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz" event={"ID":"95dd3f17-e145-49ec-8fe2-6148c3f07c80","Type":"ContainerStarted","Data":"255deffa733f6070921d532a2bbcd6bccb113337efaa216e7d3027b21b4e7357"} Nov 03 21:04:56.421710 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:56.421674 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz" Nov 03 21:04:56.422009 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:56.421988 4305 patch_prober.go:29] interesting pod/e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Nov 03 21:04:56.422090 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:56.422070 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz" podUID="95dd3f17-e145-49ec-8fe2-6148c3f07c80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Nov 03 21:04:57.422403 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:57.422358 4305 patch_prober.go:29] interesting pod/e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Nov 03 21:04:57.422889 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:57.422474 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz" podUID="95dd3f17-e145-49ec-8fe2-6148c3f07c80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Nov 03 21:04:58.422148 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:58.422106 4305 patch_prober.go:29] interesting pod/e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Nov 03 21:04:58.422353 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:58.422166 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz" podUID="95dd3f17-e145-49ec-8fe2-6148c3f07c80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Nov 03 21:04:59.422876 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:59.422835 4305 patch_prober.go:29] interesting pod/e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Nov 03 21:04:59.423293 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:04:59.422903 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz" podUID="95dd3f17-e145-49ec-8fe2-6148c3f07c80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Nov 03 21:05:00.421828 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:00.421785 4305 patch_prober.go:29] interesting pod/e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Nov 03 21:05:00.422014 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:00.421845 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz" podUID="95dd3f17-e145-49ec-8fe2-6148c3f07c80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Nov 03 21:05:01.426778 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:01.426749 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz" Nov 03 21:05:01.427300 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:01.427280 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz" Nov 03 21:05:01.428252 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:01.428231 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz" Nov 03 21:05:03.371937 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:03.371889 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:05:06.069026 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:06.068987 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f2c17e9-pt5vv"] Nov 03 21:05:06.069835 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:06.069808 4305 cpu_manager.go:414] "containerMap: removing container" podUID="3320ca8a-f6ac-44c1-a877-a62db11a904c" containerName="payload-processing" Nov 03 21:05:06.069835 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:06.069830 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3320ca8a-f6ac-44c1-a877-a62db11a904c" containerName="payload-processing" Nov 03 21:05:06.070052 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:06.069872 4305 cpu_manager.go:414] "containerMap: removing container" podUID="00169062-4b97-41c7-afa8-2a3e6f6678a3" containerName="revoke-keys" Nov 03 21:05:06.070052 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:06.069878 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00169062-4b97-41c7-afa8-2a3e6f6678a3" containerName="revoke-keys" Nov 03 21:05:06.070052 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:06.069912 4305 cpu_manager.go:414] "containerMap: removing container" podUID="47183bab-92a7-49cc-a7ef-446106ac055d" containerName="payload-pre-processing" Nov 03 21:05:06.070052 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:06.069919 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47183bab-92a7-49cc-a7ef-446106ac055d" containerName="payload-pre-processing" Nov 03 21:05:06.070249 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:06.070140 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="3320ca8a-f6ac-44c1-a877-a62db11a904c" containerName="payload-processing" Nov 03 21:05:06.070249 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:06.070162 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="00169062-4b97-41c7-afa8-2a3e6f6678a3" containerName="revoke-keys" Nov 03 21:05:06.070249 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:06.070176 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="47183bab-92a7-49cc-a7ef-446106ac055d" containerName="payload-pre-processing" Nov 03 21:05:06.076037 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:06.076011 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f2c17e9-pt5vv" Nov 03 21:05:06.102073 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:06.102040 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c7064b8-8009-416b-9437-e3e055f7bd5f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f2c17e9-pt5vv\" (UID: \"7c7064b8-8009-416b-9437-e3e055f7bd5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f2c17e9-pt5vv" Nov 03 21:05:06.102073 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:06.102075 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c7064b8-8009-416b-9437-e3e055f7bd5f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f2c17e9-pt5vv\" (UID: \"7c7064b8-8009-416b-9437-e3e055f7bd5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f2c17e9-pt5vv" Nov 03 21:05:06.102332 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:06.102310 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tldtn\" (UniqueName: \"kubernetes.io/projected/7c7064b8-8009-416b-9437-e3e055f7bd5f-kube-api-access-tldtn\") pod \"maas-api-revoke-keys-e2e-derive-6f2c17e9-pt5vv\" (UID: \"7c7064b8-8009-416b-9437-e3e055f7bd5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f2c17e9-pt5vv" Nov 03 21:05:06.132880 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:06.132844 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f2c17e9-pt5vv"] Nov 03 21:05:06.173583 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:06.173341 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:05:06.173583 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:06.173397 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:05:06.173922 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:06.173896 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77vtd\"" Nov 03 21:05:06.173979 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:06.173944 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:05:06.203710 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:06.203675 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c7064b8-8009-416b-9437-e3e055f7bd5f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f2c17e9-pt5vv\" (UID: \"7c7064b8-8009-416b-9437-e3e055f7bd5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f2c17e9-pt5vv" Nov 03 21:05:06.203710 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:06.203713 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c7064b8-8009-416b-9437-e3e055f7bd5f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f2c17e9-pt5vv\" (UID: \"7c7064b8-8009-416b-9437-e3e055f7bd5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f2c17e9-pt5vv" Nov 03 21:05:06.203997 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:06.203880 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tldtn\" (UniqueName: \"kubernetes.io/projected/7c7064b8-8009-416b-9437-e3e055f7bd5f-kube-api-access-tldtn\") pod \"maas-api-revoke-keys-e2e-derive-6f2c17e9-pt5vv\" (UID: \"7c7064b8-8009-416b-9437-e3e055f7bd5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f2c17e9-pt5vv" Nov 03 21:05:06.204340 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:06.204318 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c7064b8-8009-416b-9437-e3e055f7bd5f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f2c17e9-pt5vv\" (UID: \"7c7064b8-8009-416b-9437-e3e055f7bd5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f2c17e9-pt5vv" Nov 03 21:05:06.206303 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:06.206278 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c7064b8-8009-416b-9437-e3e055f7bd5f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-6f2c17e9-pt5vv\" (UID: \"7c7064b8-8009-416b-9437-e3e055f7bd5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f2c17e9-pt5vv" Nov 03 21:05:06.265278 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:06.265242 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tldtn\" (UniqueName: \"kubernetes.io/projected/7c7064b8-8009-416b-9437-e3e055f7bd5f-kube-api-access-tldtn\") pod \"maas-api-revoke-keys-e2e-derive-6f2c17e9-pt5vv\" (UID: \"7c7064b8-8009-416b-9437-e3e055f7bd5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f2c17e9-pt5vv" Nov 03 21:05:06.391803 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:06.391715 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f2c17e9-pt5vv" Nov 03 21:05:06.783077 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:05:06.783046 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c7064b8_8009_416b_9437_e3e055f7bd5f.slice/crio-6cf317c491066c4209ecb4fe3ae2352f404a107495672475aed74c9ee1c2a162 WatchSource:0}: Error finding container 6cf317c491066c4209ecb4fe3ae2352f404a107495672475aed74c9ee1c2a162: Status 404 returned error can't find the container with id 6cf317c491066c4209ecb4fe3ae2352f404a107495672475aed74c9ee1c2a162 Nov 03 21:05:06.784577 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:06.784553 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f2c17e9-pt5vv"] Nov 03 21:05:07.750955 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:07.750916 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f2c17e9-pt5vv" event={"ID":"7c7064b8-8009-416b-9437-e3e055f7bd5f","Type":"ContainerStarted","Data":"414f3f706b4a19d584b1da213d1240e611c35fb834c30a970397ed3e26fe038d"} Nov 03 21:05:07.751370 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:07.750977 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f2c17e9-pt5vv" event={"ID":"7c7064b8-8009-416b-9437-e3e055f7bd5f","Type":"ContainerStarted","Data":"6cf317c491066c4209ecb4fe3ae2352f404a107495672475aed74c9ee1c2a162"} Nov 03 21:05:07.871268 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:07.871208 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f2c17e9-pt5vv" podStartSLOduration=1.871190562 podStartE2EDuration="1.871190562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:05:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:05:07.865549518 +0000 UTC m=+1543.090869340" watchObservedRunningTime="2026-11-03 21:05:07.871190562 +0000 UTC m=+1543.096510352" Nov 03 21:05:08.760729 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:08.760693 4305 generic.go:356] "Generic (PLEG): container finished" podID="7c7064b8-8009-416b-9437-e3e055f7bd5f" containerID="414f3f706b4a19d584b1da213d1240e611c35fb834c30a970397ed3e26fe038d" exitCode=0 Nov 03 21:05:08.761496 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:08.760778 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f2c17e9-pt5vv" event={"ID":"7c7064b8-8009-416b-9437-e3e055f7bd5f","Type":"ContainerDied","Data":"414f3f706b4a19d584b1da213d1240e611c35fb834c30a970397ed3e26fe038d"} Nov 03 21:05:09.964876 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:09.964850 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f2c17e9-pt5vv" Nov 03 21:05:10.072089 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:10.071995 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c7064b8-8009-416b-9437-e3e055f7bd5f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c7064b8-8009-416b-9437-e3e055f7bd5f-maas-api-openshift-ca\") pod \"7c7064b8-8009-416b-9437-e3e055f7bd5f\" (UID: \"7c7064b8-8009-416b-9437-e3e055f7bd5f\") " Nov 03 21:05:10.072395 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:10.072106 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c7064b8-8009-416b-9437-e3e055f7bd5f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c7064b8-8009-416b-9437-e3e055f7bd5f-maas-api-xks-ca\") pod \"7c7064b8-8009-416b-9437-e3e055f7bd5f\" (UID: \"7c7064b8-8009-416b-9437-e3e055f7bd5f\") " Nov 03 21:05:10.072395 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:10.072184 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c7064b8-8009-416b-9437-e3e055f7bd5f-kube-api-access-tldtn\" (UniqueName: \"kubernetes.io/projected/7c7064b8-8009-416b-9437-e3e055f7bd5f-kube-api-access-tldtn\") pod \"7c7064b8-8009-416b-9437-e3e055f7bd5f\" (UID: \"7c7064b8-8009-416b-9437-e3e055f7bd5f\") " Nov 03 21:05:10.072555 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:10.072428 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c7064b8-8009-416b-9437-e3e055f7bd5f-maas-api-openshift-ca" pod "7c7064b8-8009-416b-9437-e3e055f7bd5f" (UID: "7c7064b8-8009-416b-9437-e3e055f7bd5f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:05:10.072947 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:10.072922 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c7064b8-8009-416b-9437-e3e055f7bd5f-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:05:10.074503 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:10.074483 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c7064b8-8009-416b-9437-e3e055f7bd5f-maas-api-xks-ca" pod "7c7064b8-8009-416b-9437-e3e055f7bd5f" (UID: "7c7064b8-8009-416b-9437-e3e055f7bd5f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:05:10.074560 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:10.074524 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c7064b8-8009-416b-9437-e3e055f7bd5f-kube-api-access-tldtn" pod "7c7064b8-8009-416b-9437-e3e055f7bd5f" (UID: "7c7064b8-8009-416b-9437-e3e055f7bd5f"). InnerVolumeSpecName "kube-api-access-tldtn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:05:10.174186 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:10.174146 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c7064b8-8009-416b-9437-e3e055f7bd5f-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:05:10.174186 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:10.174180 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tldtn\" (UniqueName: \"kubernetes.io/projected/7c7064b8-8009-416b-9437-e3e055f7bd5f-kube-api-access-tldtn\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:05:10.780067 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:10.780034 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f2c17e9-pt5vv" Nov 03 21:05:10.780067 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:10.780056 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f2c17e9-pt5vv" event={"ID":"7c7064b8-8009-416b-9437-e3e055f7bd5f","Type":"ContainerDied","Data":"6cf317c491066c4209ecb4fe3ae2352f404a107495672475aed74c9ee1c2a162"} Nov 03 21:05:10.780293 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:10.780108 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6cf317c491066c4209ecb4fe3ae2352f404a107495672475aed74c9ee1c2a162" Nov 03 21:05:16.324372 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:05:16.324330 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:05:17.097840 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:17.097792 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-6f2c17e9-747c59f698-dkf65"] Nov 03 21:05:17.218340 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:17.218303 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-6f2c17e9-75c5d4f87f-j6p4t"] Nov 03 21:05:22.098662 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:22.098592 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-6f2c17e9-747c59f698-dkf65" podUID="1b9f328c-f07b-4b97-ab85-0b94cbd3676c" containerName="payload-processing" containerID="cri-o://bb550ff03bf04711ca7e01d6361e40e0ed6404561cc07dc66022dbc6cee88cdf" gracePeriod=25 Nov 03 21:05:22.219546 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:22.219485 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-6f2c17e9-75c5d4f87f-j6p4t" podUID="f614f399-012f-43f3-bd18-cb62b5d089d7" containerName="payload-pre-processing" containerID="cri-o://512c1ccd3f2dfd7cc3ca159509480d6c9486d0524dd10abd71dc50e50dd08062" gracePeriod=25 Nov 03 21:05:22.406895 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:22.406866 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-6f2c17e9-747c59f698-dkf65" Nov 03 21:05:22.524149 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:22.524115 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-6f2c17e9-75c5d4f87f-j6p4t" Nov 03 21:05:22.542534 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:22.542497 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b9f328c-f07b-4b97-ab85-0b94cbd3676c-kube-api-access-9d6mc\" (UniqueName: \"kubernetes.io/projected/1b9f328c-f07b-4b97-ab85-0b94cbd3676c-kube-api-access-9d6mc\") pod \"1b9f328c-f07b-4b97-ab85-0b94cbd3676c\" (UID: \"1b9f328c-f07b-4b97-ab85-0b94cbd3676c\") " Nov 03 21:05:22.542718 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:22.542626 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b9f328c-f07b-4b97-ab85-0b94cbd3676c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b9f328c-f07b-4b97-ab85-0b94cbd3676c-plugins-config-volume\") pod \"1b9f328c-f07b-4b97-ab85-0b94cbd3676c\" (UID: \"1b9f328c-f07b-4b97-ab85-0b94cbd3676c\") " Nov 03 21:05:22.543089 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:22.543059 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b9f328c-f07b-4b97-ab85-0b94cbd3676c-plugins-config-volume" pod "1b9f328c-f07b-4b97-ab85-0b94cbd3676c" (UID: "1b9f328c-f07b-4b97-ab85-0b94cbd3676c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:05:22.543526 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:22.543499 4305 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b9f328c-f07b-4b97-ab85-0b94cbd3676c-plugins-config-volume\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:05:22.546205 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:22.546181 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b9f328c-f07b-4b97-ab85-0b94cbd3676c-kube-api-access-9d6mc" pod "1b9f328c-f07b-4b97-ab85-0b94cbd3676c" (UID: "1b9f328c-f07b-4b97-ab85-0b94cbd3676c"). InnerVolumeSpecName "kube-api-access-9d6mc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:05:22.644322 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:22.644291 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f614f399-012f-43f3-bd18-cb62b5d089d7-kube-api-access-wzs6d\" (UniqueName: \"kubernetes.io/projected/f614f399-012f-43f3-bd18-cb62b5d089d7-kube-api-access-wzs6d\") pod \"f614f399-012f-43f3-bd18-cb62b5d089d7\" (UID: \"f614f399-012f-43f3-bd18-cb62b5d089d7\") " Nov 03 21:05:22.644555 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:22.644522 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f614f399-012f-43f3-bd18-cb62b5d089d7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f614f399-012f-43f3-bd18-cb62b5d089d7-plugins-config-volume\") pod \"f614f399-012f-43f3-bd18-cb62b5d089d7\" (UID: \"f614f399-012f-43f3-bd18-cb62b5d089d7\") " Nov 03 21:05:22.644994 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:22.644969 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f614f399-012f-43f3-bd18-cb62b5d089d7-plugins-config-volume" pod "f614f399-012f-43f3-bd18-cb62b5d089d7" (UID: "f614f399-012f-43f3-bd18-cb62b5d089d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:05:22.645280 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:22.645267 4305 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f614f399-012f-43f3-bd18-cb62b5d089d7-plugins-config-volume\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:05:22.645320 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:22.645285 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d6mc\" (UniqueName: \"kubernetes.io/projected/1b9f328c-f07b-4b97-ab85-0b94cbd3676c-kube-api-access-9d6mc\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:05:22.646664 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:22.646642 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f614f399-012f-43f3-bd18-cb62b5d089d7-kube-api-access-wzs6d" pod "f614f399-012f-43f3-bd18-cb62b5d089d7" (UID: "f614f399-012f-43f3-bd18-cb62b5d089d7"). InnerVolumeSpecName "kube-api-access-wzs6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:05:22.747739 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:22.747693 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzs6d\" (UniqueName: \"kubernetes.io/projected/f614f399-012f-43f3-bd18-cb62b5d089d7-kube-api-access-wzs6d\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:05:22.917982 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:22.917944 4305 generic.go:356] "Generic (PLEG): container finished" podID="1b9f328c-f07b-4b97-ab85-0b94cbd3676c" containerID="bb550ff03bf04711ca7e01d6361e40e0ed6404561cc07dc66022dbc6cee88cdf" exitCode=0 Nov 03 21:05:22.918175 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:22.917993 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-6f2c17e9-747c59f698-dkf65" Nov 03 21:05:22.918175 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:22.918025 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-6f2c17e9-747c59f698-dkf65" event={"ID":"1b9f328c-f07b-4b97-ab85-0b94cbd3676c","Type":"ContainerDied","Data":"bb550ff03bf04711ca7e01d6361e40e0ed6404561cc07dc66022dbc6cee88cdf"} Nov 03 21:05:22.918175 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:22.918081 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-6f2c17e9-747c59f698-dkf65" event={"ID":"1b9f328c-f07b-4b97-ab85-0b94cbd3676c","Type":"ContainerDied","Data":"89560cffc0f2781fcd6b56da7f60463f4727c315f22e9bc20ae23ca264da1ea5"} Nov 03 21:05:22.918175 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:22.918107 4305 scope.go:122] "RemoveContainer" containerID="bb550ff03bf04711ca7e01d6361e40e0ed6404561cc07dc66022dbc6cee88cdf" Nov 03 21:05:22.919828 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:22.919804 4305 generic.go:356] "Generic (PLEG): container finished" podID="f614f399-012f-43f3-bd18-cb62b5d089d7" containerID="512c1ccd3f2dfd7cc3ca159509480d6c9486d0524dd10abd71dc50e50dd08062" exitCode=0 Nov 03 21:05:22.919828 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:22.919841 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-6f2c17e9-75c5d4f87f-j6p4t" Nov 03 21:05:22.920045 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:22.919868 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-6f2c17e9-75c5d4f87f-j6p4t" event={"ID":"f614f399-012f-43f3-bd18-cb62b5d089d7","Type":"ContainerDied","Data":"512c1ccd3f2dfd7cc3ca159509480d6c9486d0524dd10abd71dc50e50dd08062"} Nov 03 21:05:22.920045 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:22.919924 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-6f2c17e9-75c5d4f87f-j6p4t" event={"ID":"f614f399-012f-43f3-bd18-cb62b5d089d7","Type":"ContainerDied","Data":"6481b0004369c8e9487891df03d9e5e2712e4b80595236d3b70fde62f8f43fa9"} Nov 03 21:05:22.932170 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:22.932140 4305 scope.go:122] "RemoveContainer" containerID="bb550ff03bf04711ca7e01d6361e40e0ed6404561cc07dc66022dbc6cee88cdf" Nov 03 21:05:22.932634 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:05:22.932608 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb550ff03bf04711ca7e01d6361e40e0ed6404561cc07dc66022dbc6cee88cdf\": container with ID starting with bb550ff03bf04711ca7e01d6361e40e0ed6404561cc07dc66022dbc6cee88cdf not found: ID does not exist" containerID="bb550ff03bf04711ca7e01d6361e40e0ed6404561cc07dc66022dbc6cee88cdf" Nov 03 21:05:22.932801 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:22.932641 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb550ff03bf04711ca7e01d6361e40e0ed6404561cc07dc66022dbc6cee88cdf"} err="failed to get container status \"bb550ff03bf04711ca7e01d6361e40e0ed6404561cc07dc66022dbc6cee88cdf\": rpc error: code = NotFound desc = could not find container \"bb550ff03bf04711ca7e01d6361e40e0ed6404561cc07dc66022dbc6cee88cdf\": container with ID starting with bb550ff03bf04711ca7e01d6361e40e0ed6404561cc07dc66022dbc6cee88cdf not found: ID does not exist" Nov 03 21:05:22.932801 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:22.932660 4305 scope.go:122] "RemoveContainer" containerID="512c1ccd3f2dfd7cc3ca159509480d6c9486d0524dd10abd71dc50e50dd08062" Nov 03 21:05:22.946233 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:22.946211 4305 scope.go:122] "RemoveContainer" containerID="512c1ccd3f2dfd7cc3ca159509480d6c9486d0524dd10abd71dc50e50dd08062" Nov 03 21:05:22.946627 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:05:22.946603 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"512c1ccd3f2dfd7cc3ca159509480d6c9486d0524dd10abd71dc50e50dd08062\": container with ID starting with 512c1ccd3f2dfd7cc3ca159509480d6c9486d0524dd10abd71dc50e50dd08062 not found: ID does not exist" containerID="512c1ccd3f2dfd7cc3ca159509480d6c9486d0524dd10abd71dc50e50dd08062" Nov 03 21:05:22.946727 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:22.946634 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"512c1ccd3f2dfd7cc3ca159509480d6c9486d0524dd10abd71dc50e50dd08062"} err="failed to get container status \"512c1ccd3f2dfd7cc3ca159509480d6c9486d0524dd10abd71dc50e50dd08062\": rpc error: code = NotFound desc = could not find container \"512c1ccd3f2dfd7cc3ca159509480d6c9486d0524dd10abd71dc50e50dd08062\": container with ID starting with 512c1ccd3f2dfd7cc3ca159509480d6c9486d0524dd10abd71dc50e50dd08062 not found: ID does not exist" Nov 03 21:05:23.092661 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:23.092628 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-6f2c17e9-75c5d4f87f-j6p4t"] Nov 03 21:05:23.143389 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:23.143354 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-6f2c17e9-75c5d4f87f-j6p4t"] Nov 03 21:05:23.149843 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:23.149817 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c704c5a1-rtzf2"] Nov 03 21:05:23.156056 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:23.156032 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c704c5a1-rtzf2"] Nov 03 21:05:23.229109 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:23.229023 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-6f2c17e9-747c59f698-dkf65"] Nov 03 21:05:23.234941 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:23.234911 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-6f2c17e9-747c59f698-dkf65"] Nov 03 21:05:24.898567 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:24.898534 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b9f328c-f07b-4b97-ab85-0b94cbd3676c" path="/var/lib/kubelet/pods/1b9f328c-f07b-4b97-ab85-0b94cbd3676c/volumes" Nov 03 21:05:24.899203 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:24.899181 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2084750-79dc-4df9-bb37-7a1fff802b05" path="/var/lib/kubelet/pods/b2084750-79dc-4df9-bb37-7a1fff802b05/volumes" Nov 03 21:05:24.899692 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:24.899670 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f614f399-012f-43f3-bd18-cb62b5d089d7" path="/var/lib/kubelet/pods/f614f399-012f-43f3-bd18-cb62b5d089d7/volumes" Nov 03 21:05:25.969425 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:25.968324 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk"] Nov 03 21:05:25.970064 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:25.969558 4305 cpu_manager.go:414] "containerMap: removing container" podUID="f614f399-012f-43f3-bd18-cb62b5d089d7" containerName="payload-pre-processing" Nov 03 21:05:25.970064 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:25.969580 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f614f399-012f-43f3-bd18-cb62b5d089d7" containerName="payload-pre-processing" Nov 03 21:05:25.970064 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:25.969617 4305 cpu_manager.go:414] "containerMap: removing container" podUID="1b9f328c-f07b-4b97-ab85-0b94cbd3676c" containerName="payload-processing" Nov 03 21:05:25.970064 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:25.969625 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b9f328c-f07b-4b97-ab85-0b94cbd3676c" containerName="payload-processing" Nov 03 21:05:25.970064 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:25.969652 4305 cpu_manager.go:414] "containerMap: removing container" podUID="7c7064b8-8009-416b-9437-e3e055f7bd5f" containerName="revoke-keys" Nov 03 21:05:25.970064 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:25.969660 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c7064b8-8009-416b-9437-e3e055f7bd5f" containerName="revoke-keys" Nov 03 21:05:25.970064 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:25.969874 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="f614f399-012f-43f3-bd18-cb62b5d089d7" containerName="payload-pre-processing" Nov 03 21:05:25.970064 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:25.969890 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b9f328c-f07b-4b97-ab85-0b94cbd3676c" containerName="payload-processing" Nov 03 21:05:25.970064 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:25.969901 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c7064b8-8009-416b-9437-e3e055f7bd5f" containerName="revoke-keys" Nov 03 21:05:25.976321 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:25.976289 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk" Nov 03 21:05:26.018027 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:26.017990 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d35870fa-5e32-4297-ba1b-4f6526cf4bfd-workload-socket\") pod \"e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk\" (UID: \"d35870fa-5e32-4297-ba1b-4f6526cf4bfd\") " pod="openshift-ingress/e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk" Nov 03 21:05:26.018027 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:26.018031 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d35870fa-5e32-4297-ba1b-4f6526cf4bfd-istio-data\") pod \"e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk\" (UID: \"d35870fa-5e32-4297-ba1b-4f6526cf4bfd\") " pod="openshift-ingress/e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk" Nov 03 21:05:26.018241 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:26.018155 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d35870fa-5e32-4297-ba1b-4f6526cf4bfd-istiod-ca-cert\") pod \"e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk\" (UID: \"d35870fa-5e32-4297-ba1b-4f6526cf4bfd\") " pod="openshift-ingress/e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk" Nov 03 21:05:26.018241 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:26.018196 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d35870fa-5e32-4297-ba1b-4f6526cf4bfd-istio-token\") pod \"e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk\" (UID: \"d35870fa-5e32-4297-ba1b-4f6526cf4bfd\") " pod="openshift-ingress/e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk" Nov 03 21:05:26.018364 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:26.018343 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d35870fa-5e32-4297-ba1b-4f6526cf4bfd-istio-envoy\") pod \"e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk\" (UID: \"d35870fa-5e32-4297-ba1b-4f6526cf4bfd\") " pod="openshift-ingress/e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk" Nov 03 21:05:26.018404 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:26.018381 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d35870fa-5e32-4297-ba1b-4f6526cf4bfd-istio-podinfo\") pod \"e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk\" (UID: \"d35870fa-5e32-4297-ba1b-4f6526cf4bfd\") " pod="openshift-ingress/e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk" Nov 03 21:05:26.018497 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:26.018483 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scq7g\" (UniqueName: \"kubernetes.io/projected/d35870fa-5e32-4297-ba1b-4f6526cf4bfd-kube-api-access-scq7g\") pod \"e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk\" (UID: \"d35870fa-5e32-4297-ba1b-4f6526cf4bfd\") " pod="openshift-ingress/e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk" Nov 03 21:05:26.018559 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:26.018548 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d35870fa-5e32-4297-ba1b-4f6526cf4bfd-credential-socket\") pod \"e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk\" (UID: \"d35870fa-5e32-4297-ba1b-4f6526cf4bfd\") " pod="openshift-ingress/e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk" Nov 03 21:05:26.018601 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:26.018567 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d35870fa-5e32-4297-ba1b-4f6526cf4bfd-workload-certs\") pod \"e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk\" (UID: \"d35870fa-5e32-4297-ba1b-4f6526cf4bfd\") " pod="openshift-ingress/e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk" Nov 03 21:05:26.029050 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:26.029027 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-e41159db-openshift-default-dockercfg-spbw4\"" Nov 03 21:05:26.084923 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:26.084869 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk"] Nov 03 21:05:26.090382 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:26.090345 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac83ebbd-bxvgz"] Nov 03 21:05:26.096038 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:26.096008 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac83ebbd-bxvgz"] Nov 03 21:05:26.119760 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:26.119727 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d35870fa-5e32-4297-ba1b-4f6526cf4bfd-workload-socket\") pod \"e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk\" (UID: \"d35870fa-5e32-4297-ba1b-4f6526cf4bfd\") " pod="openshift-ingress/e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk" Nov 03 21:05:26.119760 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:26.119765 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d35870fa-5e32-4297-ba1b-4f6526cf4bfd-istio-data\") pod \"e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk\" (UID: \"d35870fa-5e32-4297-ba1b-4f6526cf4bfd\") " pod="openshift-ingress/e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk" Nov 03 21:05:26.120047 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:26.119913 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d35870fa-5e32-4297-ba1b-4f6526cf4bfd-istiod-ca-cert\") pod \"e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk\" (UID: \"d35870fa-5e32-4297-ba1b-4f6526cf4bfd\") " pod="openshift-ingress/e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk" Nov 03 21:05:26.120047 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:26.119961 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d35870fa-5e32-4297-ba1b-4f6526cf4bfd-istio-token\") pod \"e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk\" (UID: \"d35870fa-5e32-4297-ba1b-4f6526cf4bfd\") " pod="openshift-ingress/e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk" Nov 03 21:05:26.120165 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:26.120076 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d35870fa-5e32-4297-ba1b-4f6526cf4bfd-istio-envoy\") pod \"e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk\" (UID: \"d35870fa-5e32-4297-ba1b-4f6526cf4bfd\") " pod="openshift-ingress/e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk" Nov 03 21:05:26.120165 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:26.120100 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d35870fa-5e32-4297-ba1b-4f6526cf4bfd-istio-podinfo\") pod \"e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk\" (UID: \"d35870fa-5e32-4297-ba1b-4f6526cf4bfd\") " pod="openshift-ingress/e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk" Nov 03 21:05:26.120165 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:26.120157 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scq7g\" (UniqueName: \"kubernetes.io/projected/d35870fa-5e32-4297-ba1b-4f6526cf4bfd-kube-api-access-scq7g\") pod \"e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk\" (UID: \"d35870fa-5e32-4297-ba1b-4f6526cf4bfd\") " pod="openshift-ingress/e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk" Nov 03 21:05:26.120339 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:26.120226 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d35870fa-5e32-4297-ba1b-4f6526cf4bfd-workload-socket\") pod \"e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk\" (UID: \"d35870fa-5e32-4297-ba1b-4f6526cf4bfd\") " pod="openshift-ingress/e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk" Nov 03 21:05:26.120339 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:26.120262 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d35870fa-5e32-4297-ba1b-4f6526cf4bfd-credential-socket\") pod \"e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk\" (UID: \"d35870fa-5e32-4297-ba1b-4f6526cf4bfd\") " pod="openshift-ingress/e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk" Nov 03 21:05:26.120339 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:26.120271 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d35870fa-5e32-4297-ba1b-4f6526cf4bfd-istio-data\") pod \"e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk\" (UID: \"d35870fa-5e32-4297-ba1b-4f6526cf4bfd\") " pod="openshift-ingress/e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk" Nov 03 21:05:26.120339 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:26.120289 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d35870fa-5e32-4297-ba1b-4f6526cf4bfd-workload-certs\") pod \"e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk\" (UID: \"d35870fa-5e32-4297-ba1b-4f6526cf4bfd\") " pod="openshift-ingress/e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk" Nov 03 21:05:26.120612 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:26.120580 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d35870fa-5e32-4297-ba1b-4f6526cf4bfd-credential-socket\") pod \"e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk\" (UID: \"d35870fa-5e32-4297-ba1b-4f6526cf4bfd\") " pod="openshift-ingress/e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk" Nov 03 21:05:26.120667 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:26.120655 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d35870fa-5e32-4297-ba1b-4f6526cf4bfd-workload-certs\") pod \"e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk\" (UID: \"d35870fa-5e32-4297-ba1b-4f6526cf4bfd\") " pod="openshift-ingress/e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk" Nov 03 21:05:26.120723 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:26.120671 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d35870fa-5e32-4297-ba1b-4f6526cf4bfd-istiod-ca-cert\") pod \"e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk\" (UID: \"d35870fa-5e32-4297-ba1b-4f6526cf4bfd\") " pod="openshift-ingress/e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk" Nov 03 21:05:26.122548 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:26.122518 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d35870fa-5e32-4297-ba1b-4f6526cf4bfd-istio-envoy\") pod \"e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk\" (UID: \"d35870fa-5e32-4297-ba1b-4f6526cf4bfd\") " pod="openshift-ingress/e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk" Nov 03 21:05:26.122839 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:26.122817 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d35870fa-5e32-4297-ba1b-4f6526cf4bfd-istio-podinfo\") pod \"e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk\" (UID: \"d35870fa-5e32-4297-ba1b-4f6526cf4bfd\") " pod="openshift-ingress/e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk" Nov 03 21:05:26.184597 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:26.184565 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d35870fa-5e32-4297-ba1b-4f6526cf4bfd-istio-token\") pod \"e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk\" (UID: \"d35870fa-5e32-4297-ba1b-4f6526cf4bfd\") " pod="openshift-ingress/e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk" Nov 03 21:05:26.185200 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:26.185168 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scq7g\" (UniqueName: \"kubernetes.io/projected/d35870fa-5e32-4297-ba1b-4f6526cf4bfd-kube-api-access-scq7g\") pod \"e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk\" (UID: \"d35870fa-5e32-4297-ba1b-4f6526cf4bfd\") " pod="openshift-ingress/e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk" Nov 03 21:05:26.292222 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:26.292093 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk" Nov 03 21:05:26.755588 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:26.754373 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk"] Nov 03 21:05:26.757981 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:26.757937 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:05:26.758098 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:26.758012 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:05:26.758098 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:26.758040 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:05:26.902968 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:26.902930 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8378fc38-80d3-4ca3-b413-c529bab97f09" path="/var/lib/kubelet/pods/8378fc38-80d3-4ca3-b413-c529bab97f09/volumes" Nov 03 21:05:26.981126 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:26.981093 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk" event={"ID":"d35870fa-5e32-4297-ba1b-4f6526cf4bfd","Type":"ContainerStarted","Data":"74828063f3048ec2b674488e221b95d6107efb955c64ef034de43bef4eb62b9b"} Nov 03 21:05:26.981586 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:26.981148 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk" event={"ID":"d35870fa-5e32-4297-ba1b-4f6526cf4bfd","Type":"ContainerStarted","Data":"4b292bd433e26333a32ed969be0968a5974e46439f08cb34c53e93e06a7e5e5a"} Nov 03 21:05:27.110670 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:27.110548 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk" podStartSLOduration=2.110533909 podStartE2EDuration="2.110533909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:05:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:05:27.108402036 +0000 UTC m=+1562.333721829" watchObservedRunningTime="2026-11-03 21:05:27.110533909 +0000 UTC m=+1562.335853695" Nov 03 21:05:27.293346 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:27.293309 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk" Nov 03 21:05:27.322868 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:27.322826 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk" Nov 03 21:05:27.989817 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:27.989786 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk" Nov 03 21:05:27.990786 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:27.990764 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk" Nov 03 21:05:29.633634 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:29.633600 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-46590df2-tzq6b"] Nov 03 21:05:29.688320 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:29.688284 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-46590df2-tzq6b"] Nov 03 21:05:29.700950 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:29.700916 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4"] Nov 03 21:05:29.701153 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:29.701127 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4" podUID="8ecb5391-cd65-4987-8f36-95ac0f5371dd" containerName="istio-proxy" containerID="cri-o://8ae5bb6467106d4eb782f7a101be0dd69801771d59e22e8eb477027dadf97e20" gracePeriod=30 Nov 03 21:05:29.923459 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:29.922288 4305 patch_prober.go:29] interesting pod/e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:05:29.923459 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:29.922348 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4" podUID="8ecb5391-cd65-4987-8f36-95ac0f5371dd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:05:30.899272 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:30.899236 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00169062-4b97-41c7-afa8-2a3e6f6678a3" path="/var/lib/kubelet/pods/00169062-4b97-41c7-afa8-2a3e6f6678a3/volumes" Nov 03 21:05:33.976901 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:33.976863 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e41159db-6967b48544-cwjl7"] Nov 03 21:05:33.986403 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:33.984991 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e41159db-6967b48544-cwjl7" Nov 03 21:05:34.090073 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:34.090036 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-e41159db-dockercfg-hmnkg\"" Nov 03 21:05:34.090517 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:34.090481 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-e41159db\"" Nov 03 21:05:34.140231 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:34.140187 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4r96\" (UniqueName: \"kubernetes.io/projected/cd55d060-ec86-413c-8303-1d7bde266156-kube-api-access-t4r96\") pod \"payload-processing-e2e-ait-e41159db-6967b48544-cwjl7\" (UID: \"cd55d060-ec86-413c-8303-1d7bde266156\") " pod="openshift-ingress/payload-processing-e2e-ait-e41159db-6967b48544-cwjl7" Nov 03 21:05:34.140880 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:34.140851 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd55d060-ec86-413c-8303-1d7bde266156-plugins-config-volume\") pod \"payload-processing-e2e-ait-e41159db-6967b48544-cwjl7\" (UID: \"cd55d060-ec86-413c-8303-1d7bde266156\") " pod="openshift-ingress/payload-processing-e2e-ait-e41159db-6967b48544-cwjl7" Nov 03 21:05:34.222775 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:34.222649 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e41159db-6967b48544-cwjl7"] Nov 03 21:05:34.278884 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:34.251895 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd55d060-ec86-413c-8303-1d7bde266156-plugins-config-volume\") pod \"payload-processing-e2e-ait-e41159db-6967b48544-cwjl7\" (UID: \"cd55d060-ec86-413c-8303-1d7bde266156\") " pod="openshift-ingress/payload-processing-e2e-ait-e41159db-6967b48544-cwjl7" Nov 03 21:05:34.278884 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:34.252089 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4r96\" (UniqueName: \"kubernetes.io/projected/cd55d060-ec86-413c-8303-1d7bde266156-kube-api-access-t4r96\") pod \"payload-processing-e2e-ait-e41159db-6967b48544-cwjl7\" (UID: \"cd55d060-ec86-413c-8303-1d7bde266156\") " pod="openshift-ingress/payload-processing-e2e-ait-e41159db-6967b48544-cwjl7" Nov 03 21:05:34.278884 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:34.253354 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd55d060-ec86-413c-8303-1d7bde266156-plugins-config-volume\") pod \"payload-processing-e2e-ait-e41159db-6967b48544-cwjl7\" (UID: \"cd55d060-ec86-413c-8303-1d7bde266156\") " pod="openshift-ingress/payload-processing-e2e-ait-e41159db-6967b48544-cwjl7" Nov 03 21:05:34.333966 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:34.333928 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4r96\" (UniqueName: \"kubernetes.io/projected/cd55d060-ec86-413c-8303-1d7bde266156-kube-api-access-t4r96\") pod \"payload-processing-e2e-ait-e41159db-6967b48544-cwjl7\" (UID: \"cd55d060-ec86-413c-8303-1d7bde266156\") " pod="openshift-ingress/payload-processing-e2e-ait-e41159db-6967b48544-cwjl7" Nov 03 21:05:34.606454 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:34.606347 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e41159db-6967b48544-cwjl7" Nov 03 21:05:34.996790 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:34.996763 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4" Nov 03 21:05:35.027112 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:05:35.027081 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd55d060_ec86_413c_8303_1d7bde266156.slice/crio-6b5fba7333013b99d6b471c98ecab293f911e94dad679e760e8f0f8bf56afb7d WatchSource:0}: Error finding container 6b5fba7333013b99d6b471c98ecab293f911e94dad679e760e8f0f8bf56afb7d: Status 404 returned error can't find the container with id 6b5fba7333013b99d6b471c98ecab293f911e94dad679e760e8f0f8bf56afb7d Nov 03 21:05:35.030318 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:35.030295 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e41159db-6967b48544-cwjl7"] Nov 03 21:05:35.067457 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:35.067401 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ecb5391-cd65-4987-8f36-95ac0f5371dd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ecb5391-cd65-4987-8f36-95ac0f5371dd-workload-certs\") pod \"8ecb5391-cd65-4987-8f36-95ac0f5371dd\" (UID: \"8ecb5391-cd65-4987-8f36-95ac0f5371dd\") " Nov 03 21:05:35.067633 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:35.067587 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ecb5391-cd65-4987-8f36-95ac0f5371dd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ecb5391-cd65-4987-8f36-95ac0f5371dd-workload-socket\") pod \"8ecb5391-cd65-4987-8f36-95ac0f5371dd\" (UID: \"8ecb5391-cd65-4987-8f36-95ac0f5371dd\") " Nov 03 21:05:35.067676 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:35.067628 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ecb5391-cd65-4987-8f36-95ac0f5371dd-kube-api-access-mmlqh\" (UniqueName: \"kubernetes.io/projected/8ecb5391-cd65-4987-8f36-95ac0f5371dd-kube-api-access-mmlqh\") pod \"8ecb5391-cd65-4987-8f36-95ac0f5371dd\" (UID: \"8ecb5391-cd65-4987-8f36-95ac0f5371dd\") " Nov 03 21:05:35.067712 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:35.067675 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ecb5391-cd65-4987-8f36-95ac0f5371dd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ecb5391-cd65-4987-8f36-95ac0f5371dd-credential-socket\") pod \"8ecb5391-cd65-4987-8f36-95ac0f5371dd\" (UID: \"8ecb5391-cd65-4987-8f36-95ac0f5371dd\") " Nov 03 21:05:35.067752 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:35.067708 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ecb5391-cd65-4987-8f36-95ac0f5371dd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ecb5391-cd65-4987-8f36-95ac0f5371dd-istiod-ca-cert\") pod \"8ecb5391-cd65-4987-8f36-95ac0f5371dd\" (UID: \"8ecb5391-cd65-4987-8f36-95ac0f5371dd\") " Nov 03 21:05:35.067786 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:35.067749 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ecb5391-cd65-4987-8f36-95ac0f5371dd-workload-certs" pod "8ecb5391-cd65-4987-8f36-95ac0f5371dd" (UID: "8ecb5391-cd65-4987-8f36-95ac0f5371dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:35.067821 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:35.067798 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ecb5391-cd65-4987-8f36-95ac0f5371dd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ecb5391-cd65-4987-8f36-95ac0f5371dd-istio-envoy\") pod \"8ecb5391-cd65-4987-8f36-95ac0f5371dd\" (UID: \"8ecb5391-cd65-4987-8f36-95ac0f5371dd\") " Nov 03 21:05:35.067903 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:35.067866 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ecb5391-cd65-4987-8f36-95ac0f5371dd-istio-token\" (UniqueName: \"kubernetes.io/projected/8ecb5391-cd65-4987-8f36-95ac0f5371dd-istio-token\") pod \"8ecb5391-cd65-4987-8f36-95ac0f5371dd\" (UID: \"8ecb5391-cd65-4987-8f36-95ac0f5371dd\") " Nov 03 21:05:35.067903 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:35.067878 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ecb5391-cd65-4987-8f36-95ac0f5371dd-workload-socket" pod "8ecb5391-cd65-4987-8f36-95ac0f5371dd" (UID: "8ecb5391-cd65-4987-8f36-95ac0f5371dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:35.068021 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:35.067991 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ecb5391-cd65-4987-8f36-95ac0f5371dd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ecb5391-cd65-4987-8f36-95ac0f5371dd-istio-data\") pod \"8ecb5391-cd65-4987-8f36-95ac0f5371dd\" (UID: \"8ecb5391-cd65-4987-8f36-95ac0f5371dd\") " Nov 03 21:05:35.068080 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:35.067999 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ecb5391-cd65-4987-8f36-95ac0f5371dd-credential-socket" pod "8ecb5391-cd65-4987-8f36-95ac0f5371dd" (UID: "8ecb5391-cd65-4987-8f36-95ac0f5371dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:35.068171 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:35.068126 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ecb5391-cd65-4987-8f36-95ac0f5371dd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ecb5391-cd65-4987-8f36-95ac0f5371dd-istio-podinfo\") pod \"8ecb5391-cd65-4987-8f36-95ac0f5371dd\" (UID: \"8ecb5391-cd65-4987-8f36-95ac0f5371dd\") " Nov 03 21:05:35.068171 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:35.068156 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ecb5391-cd65-4987-8f36-95ac0f5371dd-istiod-ca-cert" pod "8ecb5391-cd65-4987-8f36-95ac0f5371dd" (UID: "8ecb5391-cd65-4987-8f36-95ac0f5371dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:05:35.068277 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:35.068234 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ecb5391-cd65-4987-8f36-95ac0f5371dd-istio-data" pod "8ecb5391-cd65-4987-8f36-95ac0f5371dd" (UID: "8ecb5391-cd65-4987-8f36-95ac0f5371dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:35.069533 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:35.069507 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ecb5391-cd65-4987-8f36-95ac0f5371dd-istio-data\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:05:35.069658 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:35.069534 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ecb5391-cd65-4987-8f36-95ac0f5371dd-workload-certs\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:05:35.069658 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:35.069551 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ecb5391-cd65-4987-8f36-95ac0f5371dd-workload-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:05:35.069658 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:35.069565 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ecb5391-cd65-4987-8f36-95ac0f5371dd-credential-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:05:35.069658 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:35.069579 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ecb5391-cd65-4987-8f36-95ac0f5371dd-istiod-ca-cert\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:05:35.070269 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:35.070244 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ecb5391-cd65-4987-8f36-95ac0f5371dd-istio-envoy" pod "8ecb5391-cd65-4987-8f36-95ac0f5371dd" (UID: "8ecb5391-cd65-4987-8f36-95ac0f5371dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:35.070352 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:35.070327 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ecb5391-cd65-4987-8f36-95ac0f5371dd-kube-api-access-mmlqh" pod "8ecb5391-cd65-4987-8f36-95ac0f5371dd" (UID: "8ecb5391-cd65-4987-8f36-95ac0f5371dd"). InnerVolumeSpecName "kube-api-access-mmlqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:05:35.070407 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:35.070346 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ecb5391-cd65-4987-8f36-95ac0f5371dd-istio-token" pod "8ecb5391-cd65-4987-8f36-95ac0f5371dd" (UID: "8ecb5391-cd65-4987-8f36-95ac0f5371dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:05:35.070597 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:35.070581 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ecb5391-cd65-4987-8f36-95ac0f5371dd-istio-podinfo" pod "8ecb5391-cd65-4987-8f36-95ac0f5371dd" (UID: "8ecb5391-cd65-4987-8f36-95ac0f5371dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:05:35.149518 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:35.149423 4305 generic.go:356] "Generic (PLEG): container finished" podID="8ecb5391-cd65-4987-8f36-95ac0f5371dd" containerID="8ae5bb6467106d4eb782f7a101be0dd69801771d59e22e8eb477027dadf97e20" exitCode=0 Nov 03 21:05:35.149518 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:35.149494 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4" Nov 03 21:05:35.149518 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:35.149492 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4" event={"ID":"8ecb5391-cd65-4987-8f36-95ac0f5371dd","Type":"ContainerDied","Data":"8ae5bb6467106d4eb782f7a101be0dd69801771d59e22e8eb477027dadf97e20"} Nov 03 21:05:35.149810 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:35.149547 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4" event={"ID":"8ecb5391-cd65-4987-8f36-95ac0f5371dd","Type":"ContainerDied","Data":"85f843b2fda36fc0855ec040be26b6650ff3742aa54fb7534a1ab474c22f033b"} Nov 03 21:05:35.149810 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:35.149578 4305 scope.go:122] "RemoveContainer" containerID="8ae5bb6467106d4eb782f7a101be0dd69801771d59e22e8eb477027dadf97e20" Nov 03 21:05:35.151664 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:35.151639 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e41159db-6967b48544-cwjl7" event={"ID":"cd55d060-ec86-413c-8303-1d7bde266156","Type":"ContainerStarted","Data":"8f55dcd9b33bd2860bc96e9dac6a1e017a8daeea77c363819a5e1ac956f85d24"} Nov 03 21:05:35.151793 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:35.151703 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e41159db-6967b48544-cwjl7" event={"ID":"cd55d060-ec86-413c-8303-1d7bde266156","Type":"ContainerStarted","Data":"6b5fba7333013b99d6b471c98ecab293f911e94dad679e760e8f0f8bf56afb7d"} Nov 03 21:05:35.151793 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:35.151735 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-e41159db-6967b48544-cwjl7" Nov 03 21:05:35.165054 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:35.165032 4305 scope.go:122] "RemoveContainer" containerID="53264c14ee737b1b67b6bf8dea0f1dda0bdc2fd15096e018205a2ec05dc024b4" Nov 03 21:05:35.171685 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:35.171651 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ecb5391-cd65-4987-8f36-95ac0f5371dd-istio-envoy\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:05:35.171828 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:35.171690 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ecb5391-cd65-4987-8f36-95ac0f5371dd-istio-token\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:05:35.171828 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:35.171704 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ecb5391-cd65-4987-8f36-95ac0f5371dd-istio-podinfo\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:05:35.171828 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:35.171718 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmlqh\" (UniqueName: \"kubernetes.io/projected/8ecb5391-cd65-4987-8f36-95ac0f5371dd-kube-api-access-mmlqh\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:05:35.181073 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:35.181046 4305 scope.go:122] "RemoveContainer" containerID="8ae5bb6467106d4eb782f7a101be0dd69801771d59e22e8eb477027dadf97e20" Nov 03 21:05:35.181526 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:05:35.181495 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ae5bb6467106d4eb782f7a101be0dd69801771d59e22e8eb477027dadf97e20\": container with ID starting with 8ae5bb6467106d4eb782f7a101be0dd69801771d59e22e8eb477027dadf97e20 not found: ID does not exist" containerID="8ae5bb6467106d4eb782f7a101be0dd69801771d59e22e8eb477027dadf97e20" Nov 03 21:05:35.181631 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:35.181540 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ae5bb6467106d4eb782f7a101be0dd69801771d59e22e8eb477027dadf97e20"} err="failed to get container status \"8ae5bb6467106d4eb782f7a101be0dd69801771d59e22e8eb477027dadf97e20\": rpc error: code = NotFound desc = could not find container \"8ae5bb6467106d4eb782f7a101be0dd69801771d59e22e8eb477027dadf97e20\": container with ID starting with 8ae5bb6467106d4eb782f7a101be0dd69801771d59e22e8eb477027dadf97e20 not found: ID does not exist" Nov 03 21:05:35.181631 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:35.181567 4305 scope.go:122] "RemoveContainer" containerID="53264c14ee737b1b67b6bf8dea0f1dda0bdc2fd15096e018205a2ec05dc024b4" Nov 03 21:05:35.181948 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:05:35.181927 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53264c14ee737b1b67b6bf8dea0f1dda0bdc2fd15096e018205a2ec05dc024b4\": container with ID starting with 53264c14ee737b1b67b6bf8dea0f1dda0bdc2fd15096e018205a2ec05dc024b4 not found: ID does not exist" containerID="53264c14ee737b1b67b6bf8dea0f1dda0bdc2fd15096e018205a2ec05dc024b4" Nov 03 21:05:35.182025 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:35.181956 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53264c14ee737b1b67b6bf8dea0f1dda0bdc2fd15096e018205a2ec05dc024b4"} err="failed to get container status \"53264c14ee737b1b67b6bf8dea0f1dda0bdc2fd15096e018205a2ec05dc024b4\": rpc error: code = NotFound desc = could not find container \"53264c14ee737b1b67b6bf8dea0f1dda0bdc2fd15096e018205a2ec05dc024b4\": container with ID starting with 53264c14ee737b1b67b6bf8dea0f1dda0bdc2fd15096e018205a2ec05dc024b4 not found: ID does not exist" Nov 03 21:05:35.332905 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:35.332841 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-e41159db-6967b48544-cwjl7" podStartSLOduration=2.33281995 podStartE2EDuration="2.33281995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:05:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:05:35.329031552 +0000 UTC m=+1570.554351353" watchObservedRunningTime="2026-11-03 21:05:35.33281995 +0000 UTC m=+1570.558139742" Nov 03 21:05:35.467526 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:35.458536 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4"] Nov 03 21:05:35.467526 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:35.466697 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-46590df2-gw-openshift-default-5d48ffd55c-ffdx4"] Nov 03 21:05:36.899305 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:36.899231 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ecb5391-cd65-4987-8f36-95ac0f5371dd" path="/var/lib/kubelet/pods/8ecb5391-cd65-4987-8f36-95ac0f5371dd/volumes" Nov 03 21:05:38.986940 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:38.986890 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8f1dfd9-wpnkr"] Nov 03 21:05:38.988285 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:38.988257 4305 cpu_manager.go:414] "containerMap: removing container" podUID="8ecb5391-cd65-4987-8f36-95ac0f5371dd" containerName="istio-proxy" Nov 03 21:05:38.988508 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:38.988482 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ecb5391-cd65-4987-8f36-95ac0f5371dd" containerName="istio-proxy" Nov 03 21:05:38.988648 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:38.988565 4305 cpu_manager.go:414] "containerMap: removing container" podUID="8ecb5391-cd65-4987-8f36-95ac0f5371dd" containerName="istio-proxy" Nov 03 21:05:38.988648 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:38.988573 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ecb5391-cd65-4987-8f36-95ac0f5371dd" containerName="istio-proxy" Nov 03 21:05:38.988829 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:38.988814 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ecb5391-cd65-4987-8f36-95ac0f5371dd" containerName="istio-proxy" Nov 03 21:05:38.988899 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:38.988845 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ecb5391-cd65-4987-8f36-95ac0f5371dd" containerName="istio-proxy" Nov 03 21:05:38.994757 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:38.994734 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8f1dfd9-wpnkr" Nov 03 21:05:38.996636 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:38.996611 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-04f5216c-fb66cddc-vbsb7"] Nov 03 21:05:39.002182 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:39.002157 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-04f5216c-fb66cddc-vbsb7" Nov 03 21:05:39.071457 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:39.055889 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:05:39.071457 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:39.056019 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:05:39.071457 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:39.056660 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:05:39.077458 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:39.076709 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8f1dfd9-wpnkr"] Nov 03 21:05:39.097497 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:39.087579 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77vtd\"" Nov 03 21:05:39.108993 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:39.108935 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-04f5216c-fb66cddc-vbsb7"] Nov 03 21:05:39.145594 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:39.145551 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-04f5216c-b64d9fdfb-p8dxz"] Nov 03 21:05:39.146468 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:39.146399 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3ff046c-42eb-40ce-b555-d069fc6a8567-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-04f5216c-fb66cddc-vbsb7\" (UID: \"b3ff046c-42eb-40ce-b555-d069fc6a8567\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-04f5216c-fb66cddc-vbsb7" Nov 03 21:05:39.146588 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:39.146519 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/485058fd-0a87-4d4c-aaa0-bfef9ceb39ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8f1dfd9-wpnkr\" (UID: \"485058fd-0a87-4d4c-aaa0-bfef9ceb39ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8f1dfd9-wpnkr" Nov 03 21:05:39.146588 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:39.146561 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/485058fd-0a87-4d4c-aaa0-bfef9ceb39ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8f1dfd9-wpnkr\" (UID: \"485058fd-0a87-4d4c-aaa0-bfef9ceb39ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8f1dfd9-wpnkr" Nov 03 21:05:39.146698 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:39.146644 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jdr2\" (UniqueName: \"kubernetes.io/projected/b3ff046c-42eb-40ce-b555-d069fc6a8567-kube-api-access-7jdr2\") pod \"payload-pre-processing-e2e-ait-04f5216c-fb66cddc-vbsb7\" (UID: \"b3ff046c-42eb-40ce-b555-d069fc6a8567\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-04f5216c-fb66cddc-vbsb7" Nov 03 21:05:39.146698 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:39.146675 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t9wr\" (UniqueName: \"kubernetes.io/projected/485058fd-0a87-4d4c-aaa0-bfef9ceb39ef-kube-api-access-6t9wr\") pod \"maas-api-revoke-sub-b8f1dfd9-wpnkr\" (UID: \"485058fd-0a87-4d4c-aaa0-bfef9ceb39ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8f1dfd9-wpnkr" Nov 03 21:05:39.148127 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:39.148085 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-04f5216c\"" Nov 03 21:05:39.148239 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:39.148200 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-04f5216c-dockercfg-qqpzm\"" Nov 03 21:05:39.152532 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:39.152226 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-04f5216c-b64d9fdfb-p8dxz" Nov 03 21:05:39.197302 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:39.197270 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-04f5216c-b64d9fdfb-p8dxz"] Nov 03 21:05:39.262241 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:39.254391 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e37f3c67-e93b-4933-b01e-22a99ef52532-plugins-config-volume\") pod \"payload-processing-e2e-ait-04f5216c-b64d9fdfb-p8dxz\" (UID: \"e37f3c67-e93b-4933-b01e-22a99ef52532\") " pod="openshift-ingress/payload-processing-e2e-ait-04f5216c-b64d9fdfb-p8dxz" Nov 03 21:05:39.262241 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:39.254555 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/485058fd-0a87-4d4c-aaa0-bfef9ceb39ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8f1dfd9-wpnkr\" (UID: \"485058fd-0a87-4d4c-aaa0-bfef9ceb39ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8f1dfd9-wpnkr" Nov 03 21:05:39.262241 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:39.254611 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/485058fd-0a87-4d4c-aaa0-bfef9ceb39ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8f1dfd9-wpnkr\" (UID: \"485058fd-0a87-4d4c-aaa0-bfef9ceb39ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8f1dfd9-wpnkr" Nov 03 21:05:39.262241 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:39.254710 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jdr2\" (UniqueName: \"kubernetes.io/projected/b3ff046c-42eb-40ce-b555-d069fc6a8567-kube-api-access-7jdr2\") pod \"payload-pre-processing-e2e-ait-04f5216c-fb66cddc-vbsb7\" (UID: \"b3ff046c-42eb-40ce-b555-d069fc6a8567\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-04f5216c-fb66cddc-vbsb7" Nov 03 21:05:39.262241 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:39.254753 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t9wr\" (UniqueName: \"kubernetes.io/projected/485058fd-0a87-4d4c-aaa0-bfef9ceb39ef-kube-api-access-6t9wr\") pod \"maas-api-revoke-sub-b8f1dfd9-wpnkr\" (UID: \"485058fd-0a87-4d4c-aaa0-bfef9ceb39ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8f1dfd9-wpnkr" Nov 03 21:05:39.262241 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:39.255186 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3ff046c-42eb-40ce-b555-d069fc6a8567-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-04f5216c-fb66cddc-vbsb7\" (UID: \"b3ff046c-42eb-40ce-b555-d069fc6a8567\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-04f5216c-fb66cddc-vbsb7" Nov 03 21:05:39.262241 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:39.255239 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnzbm\" (UniqueName: \"kubernetes.io/projected/e37f3c67-e93b-4933-b01e-22a99ef52532-kube-api-access-mnzbm\") pod \"payload-processing-e2e-ait-04f5216c-b64d9fdfb-p8dxz\" (UID: \"e37f3c67-e93b-4933-b01e-22a99ef52532\") " pod="openshift-ingress/payload-processing-e2e-ait-04f5216c-b64d9fdfb-p8dxz" Nov 03 21:05:39.262241 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:39.256657 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/485058fd-0a87-4d4c-aaa0-bfef9ceb39ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8f1dfd9-wpnkr\" (UID: \"485058fd-0a87-4d4c-aaa0-bfef9ceb39ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8f1dfd9-wpnkr" Nov 03 21:05:39.262241 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:39.256760 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3ff046c-42eb-40ce-b555-d069fc6a8567-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-04f5216c-fb66cddc-vbsb7\" (UID: \"b3ff046c-42eb-40ce-b555-d069fc6a8567\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-04f5216c-fb66cddc-vbsb7" Nov 03 21:05:39.283665 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:39.283626 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/485058fd-0a87-4d4c-aaa0-bfef9ceb39ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8f1dfd9-wpnkr\" (UID: \"485058fd-0a87-4d4c-aaa0-bfef9ceb39ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8f1dfd9-wpnkr" Nov 03 21:05:39.354511 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:39.349141 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jdr2\" (UniqueName: \"kubernetes.io/projected/b3ff046c-42eb-40ce-b555-d069fc6a8567-kube-api-access-7jdr2\") pod \"payload-pre-processing-e2e-ait-04f5216c-fb66cddc-vbsb7\" (UID: \"b3ff046c-42eb-40ce-b555-d069fc6a8567\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-04f5216c-fb66cddc-vbsb7" Nov 03 21:05:39.369749 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:39.369207 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t9wr\" (UniqueName: \"kubernetes.io/projected/485058fd-0a87-4d4c-aaa0-bfef9ceb39ef-kube-api-access-6t9wr\") pod \"maas-api-revoke-sub-b8f1dfd9-wpnkr\" (UID: \"485058fd-0a87-4d4c-aaa0-bfef9ceb39ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8f1dfd9-wpnkr" Nov 03 21:05:39.383410 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:39.382225 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnzbm\" (UniqueName: \"kubernetes.io/projected/e37f3c67-e93b-4933-b01e-22a99ef52532-kube-api-access-mnzbm\") pod \"payload-processing-e2e-ait-04f5216c-b64d9fdfb-p8dxz\" (UID: \"e37f3c67-e93b-4933-b01e-22a99ef52532\") " pod="openshift-ingress/payload-processing-e2e-ait-04f5216c-b64d9fdfb-p8dxz" Nov 03 21:05:39.383720 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:39.383545 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e37f3c67-e93b-4933-b01e-22a99ef52532-plugins-config-volume\") pod \"payload-processing-e2e-ait-04f5216c-b64d9fdfb-p8dxz\" (UID: \"e37f3c67-e93b-4933-b01e-22a99ef52532\") " pod="openshift-ingress/payload-processing-e2e-ait-04f5216c-b64d9fdfb-p8dxz" Nov 03 21:05:39.383720 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:39.383642 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e37f3c67-e93b-4933-b01e-22a99ef52532-plugins-config-volume\") pod \"payload-processing-e2e-ait-04f5216c-b64d9fdfb-p8dxz\" (UID: \"e37f3c67-e93b-4933-b01e-22a99ef52532\") " pod="openshift-ingress/payload-processing-e2e-ait-04f5216c-b64d9fdfb-p8dxz" Nov 03 21:05:39.447147 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:39.447102 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnzbm\" (UniqueName: \"kubernetes.io/projected/e37f3c67-e93b-4933-b01e-22a99ef52532-kube-api-access-mnzbm\") pod \"payload-processing-e2e-ait-04f5216c-b64d9fdfb-p8dxz\" (UID: \"e37f3c67-e93b-4933-b01e-22a99ef52532\") " pod="openshift-ingress/payload-processing-e2e-ait-04f5216c-b64d9fdfb-p8dxz" Nov 03 21:05:39.471818 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:39.471782 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-04f5216c-b64d9fdfb-p8dxz" Nov 03 21:05:39.625529 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:39.625265 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8f1dfd9-wpnkr" Nov 03 21:05:39.632177 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:39.632120 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-04f5216c-fb66cddc-vbsb7" Nov 03 21:05:39.870358 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:39.870330 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-04f5216c-b64d9fdfb-p8dxz"] Nov 03 21:05:39.870555 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:05:39.870535 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode37f3c67_e93b_4933_b01e_22a99ef52532.slice/crio-22d189cf0ed64d0b1d2fc5877274653df8f2ed892e3e87f93838cd65ddf546ca WatchSource:0}: Error finding container 22d189cf0ed64d0b1d2fc5877274653df8f2ed892e3e87f93838cd65ddf546ca: Status 404 returned error can't find the container with id 22d189cf0ed64d0b1d2fc5877274653df8f2ed892e3e87f93838cd65ddf546ca Nov 03 21:05:40.039035 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:05:40.039004 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3ff046c_42eb_40ce_b555_d069fc6a8567.slice/crio-3cd3da36e6f6633b16cef89adac0ffbea07fca00df18b2426614b50b61a10245 WatchSource:0}: Error finding container 3cd3da36e6f6633b16cef89adac0ffbea07fca00df18b2426614b50b61a10245: Status 404 returned error can't find the container with id 3cd3da36e6f6633b16cef89adac0ffbea07fca00df18b2426614b50b61a10245 Nov 03 21:05:40.041244 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:40.041221 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-04f5216c-fb66cddc-vbsb7"] Nov 03 21:05:40.048947 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:05:40.048914 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod485058fd_0a87_4d4c_aaa0_bfef9ceb39ef.slice/crio-c599db6c145e5686989558e61dc8351b59fec48873507a269b023145067375d5 WatchSource:0}: Error finding container c599db6c145e5686989558e61dc8351b59fec48873507a269b023145067375d5: Status 404 returned error can't find the container with id c599db6c145e5686989558e61dc8351b59fec48873507a269b023145067375d5 Nov 03 21:05:40.049095 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:40.049079 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8f1dfd9-wpnkr"] Nov 03 21:05:40.210451 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:40.210401 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-04f5216c-b64d9fdfb-p8dxz" event={"ID":"e37f3c67-e93b-4933-b01e-22a99ef52532","Type":"ContainerStarted","Data":"ce3ad58895272abc8d00cf473b893f260f5e4ba654aae6a07aec6c3bcf87424a"} Nov 03 21:05:40.210651 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:40.210479 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-04f5216c-b64d9fdfb-p8dxz" event={"ID":"e37f3c67-e93b-4933-b01e-22a99ef52532","Type":"ContainerStarted","Data":"22d189cf0ed64d0b1d2fc5877274653df8f2ed892e3e87f93838cd65ddf546ca"} Nov 03 21:05:40.210651 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:40.210510 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-04f5216c-b64d9fdfb-p8dxz" Nov 03 21:05:40.212278 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:40.212251 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8f1dfd9-wpnkr" event={"ID":"485058fd-0a87-4d4c-aaa0-bfef9ceb39ef","Type":"ContainerStarted","Data":"c599db6c145e5686989558e61dc8351b59fec48873507a269b023145067375d5"} Nov 03 21:05:40.214158 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:40.214128 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-04f5216c-fb66cddc-vbsb7" event={"ID":"b3ff046c-42eb-40ce-b555-d069fc6a8567","Type":"ContainerStarted","Data":"6ca296fa79c740386c7f49a74094eabda5e56fcbb8b111084f4f679995908839"} Nov 03 21:05:40.214291 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:40.214173 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-04f5216c-fb66cddc-vbsb7" event={"ID":"b3ff046c-42eb-40ce-b555-d069fc6a8567","Type":"ContainerStarted","Data":"3cd3da36e6f6633b16cef89adac0ffbea07fca00df18b2426614b50b61a10245"} Nov 03 21:05:40.214291 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:40.214231 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-04f5216c-fb66cddc-vbsb7" Nov 03 21:05:40.327511 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:40.327447 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-04f5216c-b64d9fdfb-p8dxz" podStartSLOduration=1.327412759 podStartE2EDuration="1.327412759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:05:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:05:40.326213741 +0000 UTC m=+1575.551533538" watchObservedRunningTime="2026-11-03 21:05:40.327412759 +0000 UTC m=+1575.552732550" Nov 03 21:05:41.226278 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:41.226244 4305 generic.go:356] "Generic (PLEG): container finished" podID="485058fd-0a87-4d4c-aaa0-bfef9ceb39ef" containerID="d3aa7304321297086ece0fc9826a71c049fc17a184f92b2c57efbe3ab84d6a0c" exitCode=0 Nov 03 21:05:41.226939 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:41.226324 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8f1dfd9-wpnkr" event={"ID":"485058fd-0a87-4d4c-aaa0-bfef9ceb39ef","Type":"ContainerDied","Data":"d3aa7304321297086ece0fc9826a71c049fc17a184f92b2c57efbe3ab84d6a0c"} Nov 03 21:05:41.352054 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:41.351999 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-04f5216c-fb66cddc-vbsb7" podStartSLOduration=3.35198404 podStartE2EDuration="3.35198404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:05:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:05:40.454884641 +0000 UTC m=+1575.680204450" watchObservedRunningTime="2026-11-03 21:05:41.35198404 +0000 UTC m=+1576.577303868" Nov 03 21:05:42.432784 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:42.432753 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8f1dfd9-wpnkr" Nov 03 21:05:42.539465 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:42.539355 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/485058fd-0a87-4d4c-aaa0-bfef9ceb39ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/485058fd-0a87-4d4c-aaa0-bfef9ceb39ef-maas-api-openshift-ca\") pod \"485058fd-0a87-4d4c-aaa0-bfef9ceb39ef\" (UID: \"485058fd-0a87-4d4c-aaa0-bfef9ceb39ef\") " Nov 03 21:05:42.539654 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:42.539640 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/485058fd-0a87-4d4c-aaa0-bfef9ceb39ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/485058fd-0a87-4d4c-aaa0-bfef9ceb39ef-maas-api-xks-ca\") pod \"485058fd-0a87-4d4c-aaa0-bfef9ceb39ef\" (UID: \"485058fd-0a87-4d4c-aaa0-bfef9ceb39ef\") " Nov 03 21:05:42.539744 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:42.539677 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/485058fd-0a87-4d4c-aaa0-bfef9ceb39ef-kube-api-access-6t9wr\" (UniqueName: \"kubernetes.io/projected/485058fd-0a87-4d4c-aaa0-bfef9ceb39ef-kube-api-access-6t9wr\") pod \"485058fd-0a87-4d4c-aaa0-bfef9ceb39ef\" (UID: \"485058fd-0a87-4d4c-aaa0-bfef9ceb39ef\") " Nov 03 21:05:42.539827 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:42.539801 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/485058fd-0a87-4d4c-aaa0-bfef9ceb39ef-maas-api-openshift-ca" pod "485058fd-0a87-4d4c-aaa0-bfef9ceb39ef" (UID: "485058fd-0a87-4d4c-aaa0-bfef9ceb39ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:05:42.540205 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:42.540186 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/485058fd-0a87-4d4c-aaa0-bfef9ceb39ef-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:05:42.541998 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:42.541960 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/485058fd-0a87-4d4c-aaa0-bfef9ceb39ef-maas-api-xks-ca" pod "485058fd-0a87-4d4c-aaa0-bfef9ceb39ef" (UID: "485058fd-0a87-4d4c-aaa0-bfef9ceb39ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:05:42.542109 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:42.542070 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/485058fd-0a87-4d4c-aaa0-bfef9ceb39ef-kube-api-access-6t9wr" pod "485058fd-0a87-4d4c-aaa0-bfef9ceb39ef" (UID: "485058fd-0a87-4d4c-aaa0-bfef9ceb39ef"). InnerVolumeSpecName "kube-api-access-6t9wr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:05:42.642037 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:42.641989 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/485058fd-0a87-4d4c-aaa0-bfef9ceb39ef-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:05:42.642037 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:42.642028 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6t9wr\" (UniqueName: \"kubernetes.io/projected/485058fd-0a87-4d4c-aaa0-bfef9ceb39ef-kube-api-access-6t9wr\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:05:43.246150 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:43.246070 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8f1dfd9-wpnkr" Nov 03 21:05:43.246150 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:43.246090 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8f1dfd9-wpnkr" event={"ID":"485058fd-0a87-4d4c-aaa0-bfef9ceb39ef","Type":"ContainerDied","Data":"c599db6c145e5686989558e61dc8351b59fec48873507a269b023145067375d5"} Nov 03 21:05:43.246150 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:43.246146 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c599db6c145e5686989558e61dc8351b59fec48873507a269b023145067375d5" Nov 03 21:05:44.126856 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:44.126814 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13e3f48a-ql8qx"] Nov 03 21:05:44.175851 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:44.175821 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13e3f48a-ql8qx"] Nov 03 21:05:44.898988 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:44.898956 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12b59755-1aa9-4448-b884-cc45ad03be96" path="/var/lib/kubelet/pods/12b59755-1aa9-4448-b884-cc45ad03be96/volumes" Nov 03 21:05:46.165357 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:46.165329 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-e41159db-6967b48544-cwjl7" Nov 03 21:05:46.329040 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:05:46.328998 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:05:47.064073 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:47.064037 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c034f31-xqldb"] Nov 03 21:05:47.069195 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:47.069171 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c034f31-xqldb"] Nov 03 21:05:48.898832 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:48.898796 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6045b8c9-a516-4e66-8421-62fe0d7fbc79" path="/var/lib/kubelet/pods/6045b8c9-a516-4e66-8421-62fe0d7fbc79/volumes" Nov 03 21:05:49.070613 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:49.070576 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1c19d30-jt2b8"] Nov 03 21:05:49.075989 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:49.075966 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1c19d30-jt2b8"] Nov 03 21:05:49.182838 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:49.182757 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:05:50.906296 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:50.906258 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1c41cc7-b059-4ce5-9c82-f02e0d0f23c0" path="/var/lib/kubelet/pods/a1c41cc7-b059-4ce5-9c82-f02e0d0f23c0/volumes" Nov 03 21:05:51.229655 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:51.229573 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-04f5216c-b64d9fdfb-p8dxz" Nov 03 21:05:51.229655 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:51.229647 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-04f5216c-fb66cddc-vbsb7" Nov 03 21:05:57.394894 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:57.394854 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f2c17e9-pt5vv"] Nov 03 21:05:57.416107 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:57.416074 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6f2c17e9-pt5vv"] Nov 03 21:05:58.908202 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:58.908164 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c7064b8-8009-416b-9437-e3e055f7bd5f" path="/var/lib/kubelet/pods/7c7064b8-8009-416b-9437-e3e055f7bd5f/volumes" Nov 03 21:05:59.620148 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:59.620114 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nw9m9"] Nov 03 21:05:59.620967 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:59.620948 4305 cpu_manager.go:414] "containerMap: removing container" podUID="485058fd-0a87-4d4c-aaa0-bfef9ceb39ef" containerName="revoke-keys" Nov 03 21:05:59.620967 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:59.620969 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="485058fd-0a87-4d4c-aaa0-bfef9ceb39ef" containerName="revoke-keys" Nov 03 21:05:59.621157 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:59.621110 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="485058fd-0a87-4d4c-aaa0-bfef9ceb39ef" containerName="revoke-keys" Nov 03 21:05:59.626507 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:59.626487 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nw9m9" Nov 03 21:05:59.679174 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:59.679150 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 21:05:59.679352 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:59.679246 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 21:05:59.679941 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:59.679926 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 21:05:59.728943 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:59.728913 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-cqrrn\"" Nov 03 21:05:59.737908 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:59.737874 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nw9m9"] Nov 03 21:05:59.771654 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:59.771607 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/929bc9f6-7a97-49d1-b866-4d6db686b0b1-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nw9m9\" (UID: \"929bc9f6-7a97-49d1-b866-4d6db686b0b1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nw9m9" Nov 03 21:05:59.771842 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:59.771687 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/929bc9f6-7a97-49d1-b866-4d6db686b0b1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nw9m9\" (UID: \"929bc9f6-7a97-49d1-b866-4d6db686b0b1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nw9m9" Nov 03 21:05:59.771842 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:59.771787 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/929bc9f6-7a97-49d1-b866-4d6db686b0b1-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nw9m9\" (UID: \"929bc9f6-7a97-49d1-b866-4d6db686b0b1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nw9m9" Nov 03 21:05:59.771842 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:59.771817 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/929bc9f6-7a97-49d1-b866-4d6db686b0b1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nw9m9\" (UID: \"929bc9f6-7a97-49d1-b866-4d6db686b0b1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nw9m9" Nov 03 21:05:59.772090 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:59.772069 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qb8v\" (UniqueName: \"kubernetes.io/projected/929bc9f6-7a97-49d1-b866-4d6db686b0b1-kube-api-access-9qb8v\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nw9m9\" (UID: \"929bc9f6-7a97-49d1-b866-4d6db686b0b1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nw9m9" Nov 03 21:05:59.772146 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:59.772126 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/929bc9f6-7a97-49d1-b866-4d6db686b0b1-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nw9m9\" (UID: \"929bc9f6-7a97-49d1-b866-4d6db686b0b1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nw9m9" Nov 03 21:05:59.772261 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:59.772241 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/929bc9f6-7a97-49d1-b866-4d6db686b0b1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nw9m9\" (UID: \"929bc9f6-7a97-49d1-b866-4d6db686b0b1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nw9m9" Nov 03 21:05:59.873738 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:59.873648 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/929bc9f6-7a97-49d1-b866-4d6db686b0b1-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nw9m9\" (UID: \"929bc9f6-7a97-49d1-b866-4d6db686b0b1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nw9m9" Nov 03 21:05:59.873738 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:59.873704 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/929bc9f6-7a97-49d1-b866-4d6db686b0b1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nw9m9\" (UID: \"929bc9f6-7a97-49d1-b866-4d6db686b0b1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nw9m9" Nov 03 21:05:59.873738 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:59.873741 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/929bc9f6-7a97-49d1-b866-4d6db686b0b1-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nw9m9\" (UID: \"929bc9f6-7a97-49d1-b866-4d6db686b0b1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nw9m9" Nov 03 21:05:59.874027 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:59.873757 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/929bc9f6-7a97-49d1-b866-4d6db686b0b1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nw9m9\" (UID: \"929bc9f6-7a97-49d1-b866-4d6db686b0b1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nw9m9" Nov 03 21:05:59.874027 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:59.873947 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qb8v\" (UniqueName: \"kubernetes.io/projected/929bc9f6-7a97-49d1-b866-4d6db686b0b1-kube-api-access-9qb8v\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nw9m9\" (UID: \"929bc9f6-7a97-49d1-b866-4d6db686b0b1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nw9m9" Nov 03 21:05:59.874027 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:59.873997 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/929bc9f6-7a97-49d1-b866-4d6db686b0b1-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nw9m9\" (UID: \"929bc9f6-7a97-49d1-b866-4d6db686b0b1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nw9m9" Nov 03 21:05:59.874172 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:59.874045 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/929bc9f6-7a97-49d1-b866-4d6db686b0b1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nw9m9\" (UID: \"929bc9f6-7a97-49d1-b866-4d6db686b0b1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nw9m9" Nov 03 21:05:59.874886 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:59.874851 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/929bc9f6-7a97-49d1-b866-4d6db686b0b1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nw9m9\" (UID: \"929bc9f6-7a97-49d1-b866-4d6db686b0b1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nw9m9" Nov 03 21:05:59.876611 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:59.876580 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/929bc9f6-7a97-49d1-b866-4d6db686b0b1-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nw9m9\" (UID: \"929bc9f6-7a97-49d1-b866-4d6db686b0b1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nw9m9" Nov 03 21:05:59.876720 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:59.876641 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/929bc9f6-7a97-49d1-b866-4d6db686b0b1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nw9m9\" (UID: \"929bc9f6-7a97-49d1-b866-4d6db686b0b1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nw9m9" Nov 03 21:05:59.876958 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:59.876936 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/929bc9f6-7a97-49d1-b866-4d6db686b0b1-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nw9m9\" (UID: \"929bc9f6-7a97-49d1-b866-4d6db686b0b1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nw9m9" Nov 03 21:05:59.876958 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:59.876949 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/929bc9f6-7a97-49d1-b866-4d6db686b0b1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nw9m9\" (UID: \"929bc9f6-7a97-49d1-b866-4d6db686b0b1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nw9m9" Nov 03 21:05:59.934053 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:59.934024 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/929bc9f6-7a97-49d1-b866-4d6db686b0b1-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nw9m9\" (UID: \"929bc9f6-7a97-49d1-b866-4d6db686b0b1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nw9m9" Nov 03 21:05:59.934988 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:59.934961 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qb8v\" (UniqueName: \"kubernetes.io/projected/929bc9f6-7a97-49d1-b866-4d6db686b0b1-kube-api-access-9qb8v\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nw9m9\" (UID: \"929bc9f6-7a97-49d1-b866-4d6db686b0b1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nw9m9" Nov 03 21:05:59.940970 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:05:59.940943 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nw9m9" Nov 03 21:06:00.389690 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:06:00.389660 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod929bc9f6_7a97_49d1_b866_4d6db686b0b1.slice/crio-3b79bbb91fae6e8f2f6e2d57c147427c2f077590d8e48c5558de6421d3aa80fd WatchSource:0}: Error finding container 3b79bbb91fae6e8f2f6e2d57c147427c2f077590d8e48c5558de6421d3aa80fd: Status 404 returned error can't find the container with id 3b79bbb91fae6e8f2f6e2d57c147427c2f077590d8e48c5558de6421d3aa80fd Nov 03 21:06:00.390462 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:00.390425 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nw9m9"] Nov 03 21:06:00.391971 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:00.391955 4305 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 21:06:00.432047 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:00.432014 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nw9m9" event={"ID":"929bc9f6-7a97-49d1-b866-4d6db686b0b1","Type":"ContainerStarted","Data":"3b79bbb91fae6e8f2f6e2d57c147427c2f077590d8e48c5558de6421d3aa80fd"} Nov 03 21:06:04.055017 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:04.054974 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b37ba63-sfzx2"] Nov 03 21:06:04.061643 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:04.061613 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b37ba63-sfzx2"] Nov 03 21:06:04.174272 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:04.174233 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:06:04.174417 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:04.174314 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:06:04.474015 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:04.473976 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nw9m9" event={"ID":"929bc9f6-7a97-49d1-b866-4d6db686b0b1","Type":"ContainerStarted","Data":"9760815d3cf7de0d9f1865b22bab6c40dfc66a18ef96c384526a8c5b5246f5a6"} Nov 03 21:06:04.474231 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:04.474045 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nw9m9" Nov 03 21:06:04.598630 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:04.598564 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nw9m9" podStartSLOduration=1.8177855109999999 podStartE2EDuration="5.598548555s" totalImagesPullingTime="3.780763044s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:05:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:06:04.593627751 +0000 UTC m=+1599.818947576" watchObservedRunningTime="2026-11-03 21:06:04.598548555 +0000 UTC m=+1599.823868345" Nov 03 21:06:04.898542 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:04.898511 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5cc19fe-8df7-4ccb-9cc6-d9c0fd429a27" path="/var/lib/kubelet/pods/b5cc19fe-8df7-4ccb-9cc6-d9c0fd429a27/volumes" Nov 03 21:06:05.488327 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:05.488281 4305 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-nw9m9 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:06:05.488923 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:05.488356 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nw9m9" podUID="929bc9f6-7a97-49d1-b866-4d6db686b0b1" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:06:06.063828 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:06.063784 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2778bcd-v2xb6"] Nov 03 21:06:06.113980 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:06.113948 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2778bcd-v2xb6"] Nov 03 21:06:06.901512 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:06.901478 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cbd2172-86f0-440f-9b21-ba958b56af86" path="/var/lib/kubelet/pods/5cbd2172-86f0-440f-9b21-ba958b56af86/volumes" Nov 03 21:06:07.544671 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:07.544637 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2ec0949-qcjkc"] Nov 03 21:06:07.560660 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:07.560626 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ec0949-qcjkc" Nov 03 21:06:07.606487 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:07.606452 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2ec0949-qcjkc"] Nov 03 21:06:07.617780 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:07.617733 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc4e2350-e601-418a-a278-05bb539d4cef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2ec0949-qcjkc\" (UID: \"dc4e2350-e601-418a-a278-05bb539d4cef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ec0949-qcjkc" Nov 03 21:06:07.618219 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:07.618189 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc4e2350-e601-418a-a278-05bb539d4cef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2ec0949-qcjkc\" (UID: \"dc4e2350-e601-418a-a278-05bb539d4cef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ec0949-qcjkc" Nov 03 21:06:07.618600 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:07.618573 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdb75\" (UniqueName: \"kubernetes.io/projected/dc4e2350-e601-418a-a278-05bb539d4cef-kube-api-access-bdb75\") pod \"maas-api-revoke-sub-f2ec0949-qcjkc\" (UID: \"dc4e2350-e601-418a-a278-05bb539d4cef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ec0949-qcjkc" Nov 03 21:06:07.648073 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:07.648038 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:06:07.648073 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:07.648079 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:06:07.648327 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:07.648117 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77vtd\"" Nov 03 21:06:07.662249 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:07.662211 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:06:07.719848 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:07.719812 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc4e2350-e601-418a-a278-05bb539d4cef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2ec0949-qcjkc\" (UID: \"dc4e2350-e601-418a-a278-05bb539d4cef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ec0949-qcjkc" Nov 03 21:06:07.720081 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:07.719919 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc4e2350-e601-418a-a278-05bb539d4cef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2ec0949-qcjkc\" (UID: \"dc4e2350-e601-418a-a278-05bb539d4cef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ec0949-qcjkc" Nov 03 21:06:07.720133 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:07.720093 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdb75\" (UniqueName: \"kubernetes.io/projected/dc4e2350-e601-418a-a278-05bb539d4cef-kube-api-access-bdb75\") pod \"maas-api-revoke-sub-f2ec0949-qcjkc\" (UID: \"dc4e2350-e601-418a-a278-05bb539d4cef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ec0949-qcjkc" Nov 03 21:06:07.720717 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:07.720692 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc4e2350-e601-418a-a278-05bb539d4cef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2ec0949-qcjkc\" (UID: \"dc4e2350-e601-418a-a278-05bb539d4cef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ec0949-qcjkc" Nov 03 21:06:07.722604 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:07.722583 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc4e2350-e601-418a-a278-05bb539d4cef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2ec0949-qcjkc\" (UID: \"dc4e2350-e601-418a-a278-05bb539d4cef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ec0949-qcjkc" Nov 03 21:06:07.781713 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:07.781678 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdb75\" (UniqueName: \"kubernetes.io/projected/dc4e2350-e601-418a-a278-05bb539d4cef-kube-api-access-bdb75\") pod \"maas-api-revoke-sub-f2ec0949-qcjkc\" (UID: \"dc4e2350-e601-418a-a278-05bb539d4cef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ec0949-qcjkc" Nov 03 21:06:07.881148 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:07.881052 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ec0949-qcjkc" Nov 03 21:06:08.272815 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:06:08.272782 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc4e2350_e601_418a_a278_05bb539d4cef.slice/crio-ee0ef90399286b0a27fab4935e942ca4fc30642a2d122c13ec0101423a6af6c1 WatchSource:0}: Error finding container ee0ef90399286b0a27fab4935e942ca4fc30642a2d122c13ec0101423a6af6c1: Status 404 returned error can't find the container with id ee0ef90399286b0a27fab4935e942ca4fc30642a2d122c13ec0101423a6af6c1 Nov 03 21:06:08.273914 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:08.273888 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2ec0949-qcjkc"] Nov 03 21:06:08.487518 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:08.487479 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nw9m9" Nov 03 21:06:08.534254 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:08.534165 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ec0949-qcjkc" event={"ID":"dc4e2350-e601-418a-a278-05bb539d4cef","Type":"ContainerStarted","Data":"ee0ef90399286b0a27fab4935e942ca4fc30642a2d122c13ec0101423a6af6c1"} Nov 03 21:06:09.546090 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:09.546057 4305 generic.go:356] "Generic (PLEG): container finished" podID="dc4e2350-e601-418a-a278-05bb539d4cef" containerID="1304695d1bf953312434c531d0ac50ecf013ac5c92257b4d2358e7afff19fd6f" exitCode=0 Nov 03 21:06:09.546663 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:09.546138 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ec0949-qcjkc" event={"ID":"dc4e2350-e601-418a-a278-05bb539d4cef","Type":"ContainerDied","Data":"1304695d1bf953312434c531d0ac50ecf013ac5c92257b4d2358e7afff19fd6f"} Nov 03 21:06:10.745835 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:10.745807 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ec0949-qcjkc" Nov 03 21:06:10.874158 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:10.874077 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc4e2350-e601-418a-a278-05bb539d4cef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc4e2350-e601-418a-a278-05bb539d4cef-maas-api-xks-ca\") pod \"dc4e2350-e601-418a-a278-05bb539d4cef\" (UID: \"dc4e2350-e601-418a-a278-05bb539d4cef\") " Nov 03 21:06:10.874331 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:10.874302 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc4e2350-e601-418a-a278-05bb539d4cef-kube-api-access-bdb75\" (UniqueName: \"kubernetes.io/projected/dc4e2350-e601-418a-a278-05bb539d4cef-kube-api-access-bdb75\") pod \"dc4e2350-e601-418a-a278-05bb539d4cef\" (UID: \"dc4e2350-e601-418a-a278-05bb539d4cef\") " Nov 03 21:06:10.874400 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:10.874376 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc4e2350-e601-418a-a278-05bb539d4cef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc4e2350-e601-418a-a278-05bb539d4cef-maas-api-openshift-ca\") pod \"dc4e2350-e601-418a-a278-05bb539d4cef\" (UID: \"dc4e2350-e601-418a-a278-05bb539d4cef\") " Nov 03 21:06:10.874837 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:10.874802 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc4e2350-e601-418a-a278-05bb539d4cef-maas-api-openshift-ca" pod "dc4e2350-e601-418a-a278-05bb539d4cef" (UID: "dc4e2350-e601-418a-a278-05bb539d4cef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:06:10.875501 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:10.875471 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc4e2350-e601-418a-a278-05bb539d4cef-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:06:10.876728 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:10.876702 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc4e2350-e601-418a-a278-05bb539d4cef-maas-api-xks-ca" pod "dc4e2350-e601-418a-a278-05bb539d4cef" (UID: "dc4e2350-e601-418a-a278-05bb539d4cef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:06:10.877950 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:10.877918 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc4e2350-e601-418a-a278-05bb539d4cef-kube-api-access-bdb75" pod "dc4e2350-e601-418a-a278-05bb539d4cef" (UID: "dc4e2350-e601-418a-a278-05bb539d4cef"). InnerVolumeSpecName "kube-api-access-bdb75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:06:10.977006 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:10.976968 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdb75\" (UniqueName: \"kubernetes.io/projected/dc4e2350-e601-418a-a278-05bb539d4cef-kube-api-access-bdb75\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:06:10.977006 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:10.977001 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc4e2350-e601-418a-a278-05bb539d4cef-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:06:11.568305 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:11.568269 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ec0949-qcjkc" Nov 03 21:06:11.568305 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:11.568272 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ec0949-qcjkc" event={"ID":"dc4e2350-e601-418a-a278-05bb539d4cef","Type":"ContainerDied","Data":"ee0ef90399286b0a27fab4935e942ca4fc30642a2d122c13ec0101423a6af6c1"} Nov 03 21:06:11.568561 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:11.568320 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee0ef90399286b0a27fab4935e942ca4fc30642a2d122c13ec0101423a6af6c1" Nov 03 21:06:16.324498 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:06:16.324459 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:06:17.057781 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:17.057744 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0c0b95e-b5sjc"] Nov 03 21:06:17.107007 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:17.106961 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0c0b95e-b5sjc"] Nov 03 21:06:18.899905 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:18.899862 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab9a9629-374b-45ef-a685-56da1ae8bfda" path="/var/lib/kubelet/pods/ab9a9629-374b-45ef-a685-56da1ae8bfda/volumes" Nov 03 21:06:23.050631 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:23.050597 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31b20f9f-bbs4g"] Nov 03 21:06:23.092489 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:23.092427 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31b20f9f-bbs4g"] Nov 03 21:06:24.899046 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:24.899009 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2806e408-4be8-4af6-90b3-4c122a06b402" path="/var/lib/kubelet/pods/2806e408-4be8-4af6-90b3-4c122a06b402/volumes" Nov 03 21:06:28.817614 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:28.817575 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:06:28.818255 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:28.818222 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd" podUID="c97844f3-937c-4873-865a-c408d83b0890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:06:28.920596 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:28.920540 4305 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" podUID="793b8ca3-3a2e-49a1-9be4-4971ba59df09" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:06:28.944491 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:28.944427 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" podUID="793b8ca3-3a2e-49a1-9be4-4971ba59df09" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 21:06:29.168591 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:29.168539 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" podUID="be06bc96-808a-46e6-9872-01baf1a11a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:06:29.193596 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:29.193539 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" podUID="63a704b4-51ce-46c3-8007-34b308aff332" containerName="main" probeResult="failure" output="Get \"https://10.134.0.101:8000/ready\": context deadline exceeded" Nov 03 21:06:29.289786 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:29.289744 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" podUID="8ee1f028-8503-4034-9954-eb06fcf402e7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": context deadline exceeded" Nov 03 21:06:29.290492 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:29.290064 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" podUID="7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:06:29.290920 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:29.290095 4305 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" podUID="6e7f30a9-0862-48a5-9ab7-e692e75bb294" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/health\": context deadline exceeded" Nov 03 21:06:29.336236 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:29.336187 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" podUID="6e7f30a9-0862-48a5-9ab7-e692e75bb294" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:06:29.481788 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:29.481589 4305 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" podUID="be06bc96-808a-46e6-9872-01baf1a11a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.107:8000/health\": context deadline exceeded" Nov 03 21:06:29.565525 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:29.564077 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" podUID="7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:06:29.943575 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:29.940043 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" podUID="006c5eae-06af-447b-a1d8-cd719cce2a92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:06:29.948023 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:29.947979 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" podUID="793b8ca3-3a2e-49a1-9be4-4971ba59df09" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 03 21:06:29.949159 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:29.948830 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" Nov 03 21:06:29.980645 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:29.980598 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" podUID="d5d3dcaa-a51a-40c6-8049-871b300552e1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:06:30.072470 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:30.061407 4305 scope.go:122] "RemoveContainer" containerID="f3137d3590b804d50dcda25a28ebab28d2413e0900b7f35a75774eb8647a3104" Nov 03 21:06:30.172631 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:30.172566 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" podUID="be06bc96-808a-46e6-9872-01baf1a11a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:06:30.206560 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:30.193757 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" Nov 03 21:06:30.206560 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:30.194188 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" podUID="63a704b4-51ce-46c3-8007-34b308aff332" containerName="main" probeResult="failure" output="Get \"https://10.134.0.101:8000/ready\": context deadline exceeded" Nov 03 21:06:30.206560 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:30.196850 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" Nov 03 21:06:30.286221 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:30.286192 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" Nov 03 21:06:30.312829 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:30.302679 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" Nov 03 21:06:30.312829 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:30.307001 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" Nov 03 21:06:30.312829 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:30.307514 4305 scope.go:122] "RemoveContainer" containerID="1ac02e123f643510b8d45f00a609596448b3e421ee7b35d987619ef6897a2627" Nov 03 21:06:30.312829 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:30.307507 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" podUID="7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 03 21:06:30.312829 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:30.307805 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" Nov 03 21:06:30.617636 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:30.617514 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" Nov 03 21:06:32.056902 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:32.056866 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ffa5bc4-2lq5j"] Nov 03 21:06:32.108399 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:32.108357 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ffa5bc4-2lq5j"] Nov 03 21:06:32.898139 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:32.898105 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce39a4a7-549b-4163-aab1-ac6581c9c451" path="/var/lib/kubelet/pods/ce39a4a7-549b-4163-aab1-ac6581c9c451/volumes" Nov 03 21:06:41.054930 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:41.054892 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0dfa3141-kbdr2"] Nov 03 21:06:41.103974 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:41.103939 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0dfa3141-kbdr2"] Nov 03 21:06:42.898521 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:42.898488 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58c57b4a-f8ee-4c2a-b83a-84ec067f0be4" path="/var/lib/kubelet/pods/58c57b4a-f8ee-4c2a-b83a-84ec067f0be4/volumes" Nov 03 21:06:46.050944 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:46.050910 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ded6da31-znppf"] Nov 03 21:06:46.100411 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:46.100379 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ded6da31-znppf"] Nov 03 21:06:46.328887 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:06:46.328800 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:06:46.899812 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:46.899712 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a822cf6-57c9-4941-9797-e4605586b4b0" path="/var/lib/kubelet/pods/0a822cf6-57c9-4941-9797-e4605586b4b0/volumes" Nov 03 21:06:47.063811 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:47.063776 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07445efa-87rzl"] Nov 03 21:06:47.112802 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:47.112765 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07445efa-87rzl"] Nov 03 21:06:48.898288 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:48.898250 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50254126-8dc4-46f1-9a58-847f61765f36" path="/var/lib/kubelet/pods/50254126-8dc4-46f1-9a58-847f61765f36/volumes" Nov 03 21:06:50.465211 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:50.465168 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e3917193-mpsw2"] Nov 03 21:06:50.465977 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:50.465962 4305 cpu_manager.go:414] "containerMap: removing container" podUID="dc4e2350-e601-418a-a278-05bb539d4cef" containerName="revoke-keys" Nov 03 21:06:50.466029 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:50.465978 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc4e2350-e601-418a-a278-05bb539d4cef" containerName="revoke-keys" Nov 03 21:06:50.466110 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:50.466100 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc4e2350-e601-418a-a278-05bb539d4cef" containerName="revoke-keys" Nov 03 21:06:50.470141 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:50.470125 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e3917193-mpsw2" Nov 03 21:06:50.526354 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:50.526323 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e3917193-mpsw2"] Nov 03 21:06:50.570047 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:50.570015 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:06:50.570249 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:50.570020 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:06:50.570249 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:50.570054 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:06:50.570249 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:50.570136 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77vtd\"" Nov 03 21:06:50.598363 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:50.598323 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9svjw\" (UniqueName: \"kubernetes.io/projected/a170fb6b-d3eb-474a-a4cc-1fced5daf1c5-kube-api-access-9svjw\") pod \"maas-api-revoke-keys-e2e-mt-e3917193-mpsw2\" (UID: \"a170fb6b-d3eb-474a-a4cc-1fced5daf1c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e3917193-mpsw2" Nov 03 21:06:50.598627 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:50.598579 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a170fb6b-d3eb-474a-a4cc-1fced5daf1c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-e3917193-mpsw2\" (UID: \"a170fb6b-d3eb-474a-a4cc-1fced5daf1c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e3917193-mpsw2" Nov 03 21:06:50.598794 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:50.598774 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a170fb6b-d3eb-474a-a4cc-1fced5daf1c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-e3917193-mpsw2\" (UID: \"a170fb6b-d3eb-474a-a4cc-1fced5daf1c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e3917193-mpsw2" Nov 03 21:06:50.700321 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:50.700284 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a170fb6b-d3eb-474a-a4cc-1fced5daf1c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-e3917193-mpsw2\" (UID: \"a170fb6b-d3eb-474a-a4cc-1fced5daf1c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e3917193-mpsw2" Nov 03 21:06:50.700611 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:50.700427 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9svjw\" (UniqueName: \"kubernetes.io/projected/a170fb6b-d3eb-474a-a4cc-1fced5daf1c5-kube-api-access-9svjw\") pod \"maas-api-revoke-keys-e2e-mt-e3917193-mpsw2\" (UID: \"a170fb6b-d3eb-474a-a4cc-1fced5daf1c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e3917193-mpsw2" Nov 03 21:06:50.700611 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:50.700529 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a170fb6b-d3eb-474a-a4cc-1fced5daf1c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-e3917193-mpsw2\" (UID: \"a170fb6b-d3eb-474a-a4cc-1fced5daf1c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e3917193-mpsw2" Nov 03 21:06:50.701025 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:50.700999 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a170fb6b-d3eb-474a-a4cc-1fced5daf1c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-e3917193-mpsw2\" (UID: \"a170fb6b-d3eb-474a-a4cc-1fced5daf1c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e3917193-mpsw2" Nov 03 21:06:50.703331 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:50.703304 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a170fb6b-d3eb-474a-a4cc-1fced5daf1c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-e3917193-mpsw2\" (UID: \"a170fb6b-d3eb-474a-a4cc-1fced5daf1c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e3917193-mpsw2" Nov 03 21:06:50.761463 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:50.761372 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9svjw\" (UniqueName: \"kubernetes.io/projected/a170fb6b-d3eb-474a-a4cc-1fced5daf1c5-kube-api-access-9svjw\") pod \"maas-api-revoke-keys-e2e-mt-e3917193-mpsw2\" (UID: \"a170fb6b-d3eb-474a-a4cc-1fced5daf1c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e3917193-mpsw2" Nov 03 21:06:50.786344 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:50.786310 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e3917193-mpsw2" Nov 03 21:06:51.051143 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:51.051052 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-077e9e36-5jmj4"] Nov 03 21:06:51.056474 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:51.056424 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-077e9e36-5jmj4"] Nov 03 21:06:51.174685 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:06:51.174647 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda170fb6b_d3eb_474a_a4cc_1fced5daf1c5.slice/crio-c6c4c028253b39a5b0135db0d680299fabaa77719ac58f9f1a32a05642daa788 WatchSource:0}: Error finding container c6c4c028253b39a5b0135db0d680299fabaa77719ac58f9f1a32a05642daa788: Status 404 returned error can't find the container with id c6c4c028253b39a5b0135db0d680299fabaa77719ac58f9f1a32a05642daa788 Nov 03 21:06:51.174855 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:51.174834 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e3917193-mpsw2"] Nov 03 21:06:52.029171 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:52.029140 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e3917193-mpsw2" event={"ID":"a170fb6b-d3eb-474a-a4cc-1fced5daf1c5","Type":"ContainerStarted","Data":"dda29973d6a71c376a65089459088e34aed649e321c091853162af37b5a64abf"} Nov 03 21:06:52.029628 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:52.029196 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e3917193-mpsw2" event={"ID":"a170fb6b-d3eb-474a-a4cc-1fced5daf1c5","Type":"ContainerStarted","Data":"c6c4c028253b39a5b0135db0d680299fabaa77719ac58f9f1a32a05642daa788"} Nov 03 21:06:52.148226 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:52.148163 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e3917193-mpsw2" podStartSLOduration=2.148148799 podStartE2EDuration="2.148148799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:06:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:06:52.144986686 +0000 UTC m=+1647.370306499" watchObservedRunningTime="2026-11-03 21:06:52.148148799 +0000 UTC m=+1647.373468589" Nov 03 21:06:52.899266 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:52.899229 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e2d33ae-aad0-4600-a817-5c6f6d6771f7" path="/var/lib/kubelet/pods/9e2d33ae-aad0-4600-a817-5c6f6d6771f7/volumes" Nov 03 21:06:53.040160 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:53.040123 4305 generic.go:356] "Generic (PLEG): container finished" podID="a170fb6b-d3eb-474a-a4cc-1fced5daf1c5" containerID="dda29973d6a71c376a65089459088e34aed649e321c091853162af37b5a64abf" exitCode=0 Nov 03 21:06:53.040605 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:53.040217 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e3917193-mpsw2" event={"ID":"a170fb6b-d3eb-474a-a4cc-1fced5daf1c5","Type":"ContainerDied","Data":"dda29973d6a71c376a65089459088e34aed649e321c091853162af37b5a64abf"} Nov 03 21:06:54.259547 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:54.259523 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e3917193-mpsw2" Nov 03 21:06:54.360398 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:54.360298 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a170fb6b-d3eb-474a-a4cc-1fced5daf1c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a170fb6b-d3eb-474a-a4cc-1fced5daf1c5-maas-api-xks-ca\") pod \"a170fb6b-d3eb-474a-a4cc-1fced5daf1c5\" (UID: \"a170fb6b-d3eb-474a-a4cc-1fced5daf1c5\") " Nov 03 21:06:54.360641 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:54.360542 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a170fb6b-d3eb-474a-a4cc-1fced5daf1c5-kube-api-access-9svjw\" (UniqueName: \"kubernetes.io/projected/a170fb6b-d3eb-474a-a4cc-1fced5daf1c5-kube-api-access-9svjw\") pod \"a170fb6b-d3eb-474a-a4cc-1fced5daf1c5\" (UID: \"a170fb6b-d3eb-474a-a4cc-1fced5daf1c5\") " Nov 03 21:06:54.360834 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:54.360809 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a170fb6b-d3eb-474a-a4cc-1fced5daf1c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a170fb6b-d3eb-474a-a4cc-1fced5daf1c5-maas-api-openshift-ca\") pod \"a170fb6b-d3eb-474a-a4cc-1fced5daf1c5\" (UID: \"a170fb6b-d3eb-474a-a4cc-1fced5daf1c5\") " Nov 03 21:06:54.361344 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:54.361315 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a170fb6b-d3eb-474a-a4cc-1fced5daf1c5-maas-api-openshift-ca" pod "a170fb6b-d3eb-474a-a4cc-1fced5daf1c5" (UID: "a170fb6b-d3eb-474a-a4cc-1fced5daf1c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:06:54.361832 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:54.361809 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a170fb6b-d3eb-474a-a4cc-1fced5daf1c5-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:06:54.362833 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:54.362806 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a170fb6b-d3eb-474a-a4cc-1fced5daf1c5-maas-api-xks-ca" pod "a170fb6b-d3eb-474a-a4cc-1fced5daf1c5" (UID: "a170fb6b-d3eb-474a-a4cc-1fced5daf1c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:06:54.362930 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:54.362855 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a170fb6b-d3eb-474a-a4cc-1fced5daf1c5-kube-api-access-9svjw" pod "a170fb6b-d3eb-474a-a4cc-1fced5daf1c5" (UID: "a170fb6b-d3eb-474a-a4cc-1fced5daf1c5"). InnerVolumeSpecName "kube-api-access-9svjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:06:54.463509 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:54.463475 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a170fb6b-d3eb-474a-a4cc-1fced5daf1c5-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:06:54.463509 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:54.463506 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9svjw\" (UniqueName: \"kubernetes.io/projected/a170fb6b-d3eb-474a-a4cc-1fced5daf1c5-kube-api-access-9svjw\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:06:55.060009 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:55.059972 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e3917193-mpsw2" Nov 03 21:06:55.060009 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:55.059983 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e3917193-mpsw2" event={"ID":"a170fb6b-d3eb-474a-a4cc-1fced5daf1c5","Type":"ContainerDied","Data":"c6c4c028253b39a5b0135db0d680299fabaa77719ac58f9f1a32a05642daa788"} Nov 03 21:06:55.060234 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:55.060041 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6c4c028253b39a5b0135db0d680299fabaa77719ac58f9f1a32a05642daa788" Nov 03 21:06:56.990928 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:56.990896 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:06:57.053094 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:57.053052 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14b52bda-77lqz"] Nov 03 21:06:57.060219 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:57.060187 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14b52bda-77lqz"] Nov 03 21:06:58.971402 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:06:58.971365 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fc27cc8-cbb7-4c99-95f7-1678fa27b72d" path="/var/lib/kubelet/pods/2fc27cc8-cbb7-4c99-95f7-1678fa27b72d/volumes" Nov 03 21:07:00.326462 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:00.319560 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" podUID="793b8ca3-3a2e-49a1-9be4-4971ba59df09" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:07:00.355463 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:00.340563 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" podUID="6e7f30a9-0862-48a5-9ab7-e692e75bb294" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 03 21:07:00.676458 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:00.675552 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" podUID="7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:07:00.935521 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:00.934559 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" podUID="006c5eae-06af-447b-a1d8-cd719cce2a92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:07:00.957636 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:00.957591 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" podUID="d5d3dcaa-a51a-40c6-8049-871b300552e1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded" Nov 03 21:07:01.126579 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:01.126502 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fad5eaf-7tg2k"] Nov 03 21:07:01.170245 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:01.170202 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fad5eaf-7tg2k"] Nov 03 21:07:01.271762 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:01.271659 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" podUID="8793f095-97b1-4e0a-8382-b541f90a0de2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:07:01.298742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:01.298694 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" podUID="63a704b4-51ce-46c3-8007-34b308aff332" containerName="main" probeResult="failure" output="Get \"https://10.134.0.101:8000/ready\": context deadline exceeded" Nov 03 21:07:01.299933 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:01.299892 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" podUID="80de7944-70f6-49e1-a0e8-4b82d5caad0d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:07:01.300462 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:01.300122 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" podUID="8ee1f028-8503-4034-9954-eb06fcf402e7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": context deadline exceeded" Nov 03 21:07:01.305552 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:01.305517 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" podUID="be06bc96-808a-46e6-9872-01baf1a11a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:07:01.322584 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:01.322537 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" podUID="793b8ca3-3a2e-49a1-9be4-4971ba59df09" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Nov 03 21:07:01.323264 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:01.323238 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" Nov 03 21:07:01.345286 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:01.345246 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" podUID="6e7f30a9-0862-48a5-9ab7-e692e75bb294" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:07:01.346271 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:01.346248 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" Nov 03 21:07:01.746460 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:01.741478 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" podUID="7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:07:01.746460 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:01.741874 4305 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" podUID="006c5eae-06af-447b-a1d8-cd719cce2a92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:07:01.771461 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:01.754617 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" podUID="7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 03 21:07:01.771461 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:01.755051 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" Nov 03 21:07:01.936219 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:01.936161 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" podUID="006c5eae-06af-447b-a1d8-cd719cce2a92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:07:01.970962 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:01.970931 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" Nov 03 21:07:02.322469 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:02.321963 4305 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" podUID="d5d3dcaa-a51a-40c6-8049-871b300552e1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:07:02.347464 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:02.339555 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" podUID="793b8ca3-3a2e-49a1-9be4-4971ba59df09" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:07:02.397533 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:02.351130 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" podUID="6e7f30a9-0862-48a5-9ab7-e692e75bb294" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:07:02.397533 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:02.372715 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" podUID="80de7944-70f6-49e1-a0e8-4b82d5caad0d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:07:02.397533 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:02.373046 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" podUID="8ee1f028-8503-4034-9954-eb06fcf402e7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.93:8000/ready\": context deadline exceeded" Nov 03 21:07:02.403413 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:02.401655 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" Nov 03 21:07:02.403413 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:02.401761 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" Nov 03 21:07:02.405078 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:02.403981 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" podUID="8793f095-97b1-4e0a-8382-b541f90a0de2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:07:02.406375 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:02.405883 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" podUID="be06bc96-808a-46e6-9872-01baf1a11a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.107:8000/ready\": context deadline exceeded" Nov 03 21:07:02.444830 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:02.444796 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" Nov 03 21:07:02.479095 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:02.479057 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" Nov 03 21:07:02.504421 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:02.450521 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" podUID="63a704b4-51ce-46c3-8007-34b308aff332" containerName="main" probeResult="failure" output="Get \"https://10.134.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:07:02.534315 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:02.474719 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" podUID="d5d3dcaa-a51a-40c6-8049-871b300552e1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded" Nov 03 21:07:02.573663 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:02.570279 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" Nov 03 21:07:02.573663 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:02.570375 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" Nov 03 21:07:02.752453 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:02.743606 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" podUID="7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:07:02.813115 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:02.813067 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" Nov 03 21:07:02.813728 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:02.813701 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" Nov 03 21:07:02.824392 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:02.824305 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" podUID="7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:07:02.829423 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:02.829397 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw" Nov 03 21:07:02.830361 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:02.830326 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" Nov 03 21:07:02.840664 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:02.840640 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" Nov 03 21:07:02.840896 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:02.840884 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e3917193-68c654f4dd-ttdqc"] Nov 03 21:07:02.841066 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:02.841057 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" Nov 03 21:07:02.841180 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:02.841171 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e3917193-665777fcf9-khvcd"] Nov 03 21:07:02.841329 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:02.841313 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" Nov 03 21:07:02.841470 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:02.841459 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" Nov 03 21:07:02.841594 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:02.841586 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" Nov 03 21:07:02.841720 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:02.841712 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" Nov 03 21:07:02.890732 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:02.890700 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" Nov 03 21:07:02.941365 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:02.940975 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4c6f39d-5dcd-49a9-967b-4303f59d75c0" path="/var/lib/kubelet/pods/d4c6f39d-5dcd-49a9-967b-4303f59d75c0/volumes" Nov 03 21:07:02.941365 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:02.941367 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" Nov 03 21:07:07.844747 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:07.844380 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-e3917193-68c654f4dd-ttdqc" podUID="facfcb94-b6b7-4072-b4f0-49ef05e5dde8" containerName="payload-processing" containerID="cri-o://20d35ec6d07ff7fb4194128f42ebbc5cc3bce3071306d400ab7770409f9c194e" gracePeriod=25 Nov 03 21:07:07.844747 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:07.844391 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-e3917193-665777fcf9-khvcd" podUID="998a67c4-3f64-4922-a2c6-4b7629c96b28" containerName="payload-pre-processing" containerID="cri-o://44c8a7d9abe1fbe91326914c43322962425f9f303191432c48298c80cfa54134" gracePeriod=25 Nov 03 21:07:08.206312 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:08.206284 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e3917193-68c654f4dd-ttdqc" Nov 03 21:07:08.210577 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:08.210552 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e3917193-665777fcf9-khvcd" Nov 03 21:07:08.299890 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:08.299850 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/998a67c4-3f64-4922-a2c6-4b7629c96b28-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/998a67c4-3f64-4922-a2c6-4b7629c96b28-plugins-config-volume\") pod \"998a67c4-3f64-4922-a2c6-4b7629c96b28\" (UID: \"998a67c4-3f64-4922-a2c6-4b7629c96b28\") " Nov 03 21:07:08.300104 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:08.300086 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/998a67c4-3f64-4922-a2c6-4b7629c96b28-kube-api-access-mg4ml\" (UniqueName: \"kubernetes.io/projected/998a67c4-3f64-4922-a2c6-4b7629c96b28-kube-api-access-mg4ml\") pod \"998a67c4-3f64-4922-a2c6-4b7629c96b28\" (UID: \"998a67c4-3f64-4922-a2c6-4b7629c96b28\") " Nov 03 21:07:08.300200 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:08.300158 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/facfcb94-b6b7-4072-b4f0-49ef05e5dde8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/facfcb94-b6b7-4072-b4f0-49ef05e5dde8-plugins-config-volume\") pod \"facfcb94-b6b7-4072-b4f0-49ef05e5dde8\" (UID: \"facfcb94-b6b7-4072-b4f0-49ef05e5dde8\") " Nov 03 21:07:08.300615 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:08.300342 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/facfcb94-b6b7-4072-b4f0-49ef05e5dde8-kube-api-access-tkgcr\" (UniqueName: \"kubernetes.io/projected/facfcb94-b6b7-4072-b4f0-49ef05e5dde8-kube-api-access-tkgcr\") pod \"facfcb94-b6b7-4072-b4f0-49ef05e5dde8\" (UID: \"facfcb94-b6b7-4072-b4f0-49ef05e5dde8\") " Nov 03 21:07:08.300615 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:08.300484 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/998a67c4-3f64-4922-a2c6-4b7629c96b28-plugins-config-volume" pod "998a67c4-3f64-4922-a2c6-4b7629c96b28" (UID: "998a67c4-3f64-4922-a2c6-4b7629c96b28"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:07:08.300615 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:08.300570 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/facfcb94-b6b7-4072-b4f0-49ef05e5dde8-plugins-config-volume" pod "facfcb94-b6b7-4072-b4f0-49ef05e5dde8" (UID: "facfcb94-b6b7-4072-b4f0-49ef05e5dde8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:07:08.301057 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:08.301036 4305 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/998a67c4-3f64-4922-a2c6-4b7629c96b28-plugins-config-volume\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:07:08.301138 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:08.301061 4305 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/facfcb94-b6b7-4072-b4f0-49ef05e5dde8-plugins-config-volume\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:07:08.302560 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:08.302534 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/998a67c4-3f64-4922-a2c6-4b7629c96b28-kube-api-access-mg4ml" pod "998a67c4-3f64-4922-a2c6-4b7629c96b28" (UID: "998a67c4-3f64-4922-a2c6-4b7629c96b28"). InnerVolumeSpecName "kube-api-access-mg4ml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:07:08.302628 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:08.302594 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/facfcb94-b6b7-4072-b4f0-49ef05e5dde8-kube-api-access-tkgcr" pod "facfcb94-b6b7-4072-b4f0-49ef05e5dde8" (UID: "facfcb94-b6b7-4072-b4f0-49ef05e5dde8"). InnerVolumeSpecName "kube-api-access-tkgcr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:07:08.402810 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:08.402729 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg4ml\" (UniqueName: \"kubernetes.io/projected/998a67c4-3f64-4922-a2c6-4b7629c96b28-kube-api-access-mg4ml\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:07:08.402810 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:08.402759 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkgcr\" (UniqueName: \"kubernetes.io/projected/facfcb94-b6b7-4072-b4f0-49ef05e5dde8-kube-api-access-tkgcr\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:07:08.939811 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:08.939773 4305 generic.go:356] "Generic (PLEG): container finished" podID="998a67c4-3f64-4922-a2c6-4b7629c96b28" containerID="44c8a7d9abe1fbe91326914c43322962425f9f303191432c48298c80cfa54134" exitCode=0 Nov 03 21:07:08.940420 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:08.939851 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e3917193-665777fcf9-khvcd" Nov 03 21:07:08.940420 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:08.939857 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e3917193-665777fcf9-khvcd" event={"ID":"998a67c4-3f64-4922-a2c6-4b7629c96b28","Type":"ContainerDied","Data":"44c8a7d9abe1fbe91326914c43322962425f9f303191432c48298c80cfa54134"} Nov 03 21:07:08.940420 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:08.939939 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e3917193-665777fcf9-khvcd" event={"ID":"998a67c4-3f64-4922-a2c6-4b7629c96b28","Type":"ContainerDied","Data":"a99795044dded5fe0a9848791d2de1fe1664ff8cfd950c9370aaa675447a4962"} Nov 03 21:07:08.940420 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:08.939970 4305 scope.go:122] "RemoveContainer" containerID="44c8a7d9abe1fbe91326914c43322962425f9f303191432c48298c80cfa54134" Nov 03 21:07:08.944495 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:08.944116 4305 generic.go:356] "Generic (PLEG): container finished" podID="facfcb94-b6b7-4072-b4f0-49ef05e5dde8" containerID="20d35ec6d07ff7fb4194128f42ebbc5cc3bce3071306d400ab7770409f9c194e" exitCode=0 Nov 03 21:07:08.944495 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:08.944267 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e3917193-68c654f4dd-ttdqc" event={"ID":"facfcb94-b6b7-4072-b4f0-49ef05e5dde8","Type":"ContainerDied","Data":"20d35ec6d07ff7fb4194128f42ebbc5cc3bce3071306d400ab7770409f9c194e"} Nov 03 21:07:08.944495 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:08.944310 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e3917193-68c654f4dd-ttdqc" Nov 03 21:07:08.944495 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:08.944313 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e3917193-68c654f4dd-ttdqc" event={"ID":"facfcb94-b6b7-4072-b4f0-49ef05e5dde8","Type":"ContainerDied","Data":"58eec6978b0b123f990592be7f586680a451025861e25fcc9954b42ea9a2e6cf"} Nov 03 21:07:08.956810 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:08.956785 4305 scope.go:122] "RemoveContainer" containerID="44c8a7d9abe1fbe91326914c43322962425f9f303191432c48298c80cfa54134" Nov 03 21:07:08.957151 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:07:08.957130 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44c8a7d9abe1fbe91326914c43322962425f9f303191432c48298c80cfa54134\": container with ID starting with 44c8a7d9abe1fbe91326914c43322962425f9f303191432c48298c80cfa54134 not found: ID does not exist" containerID="44c8a7d9abe1fbe91326914c43322962425f9f303191432c48298c80cfa54134" Nov 03 21:07:08.957226 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:08.957161 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44c8a7d9abe1fbe91326914c43322962425f9f303191432c48298c80cfa54134"} err="failed to get container status \"44c8a7d9abe1fbe91326914c43322962425f9f303191432c48298c80cfa54134\": rpc error: code = NotFound desc = could not find container \"44c8a7d9abe1fbe91326914c43322962425f9f303191432c48298c80cfa54134\": container with ID starting with 44c8a7d9abe1fbe91326914c43322962425f9f303191432c48298c80cfa54134 not found: ID does not exist" Nov 03 21:07:08.957226 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:08.957180 4305 scope.go:122] "RemoveContainer" containerID="20d35ec6d07ff7fb4194128f42ebbc5cc3bce3071306d400ab7770409f9c194e" Nov 03 21:07:09.030372 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:09.030343 4305 scope.go:122] "RemoveContainer" containerID="20d35ec6d07ff7fb4194128f42ebbc5cc3bce3071306d400ab7770409f9c194e" Nov 03 21:07:09.030740 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:07:09.030719 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20d35ec6d07ff7fb4194128f42ebbc5cc3bce3071306d400ab7770409f9c194e\": container with ID starting with 20d35ec6d07ff7fb4194128f42ebbc5cc3bce3071306d400ab7770409f9c194e not found: ID does not exist" containerID="20d35ec6d07ff7fb4194128f42ebbc5cc3bce3071306d400ab7770409f9c194e" Nov 03 21:07:09.030811 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:09.030751 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20d35ec6d07ff7fb4194128f42ebbc5cc3bce3071306d400ab7770409f9c194e"} err="failed to get container status \"20d35ec6d07ff7fb4194128f42ebbc5cc3bce3071306d400ab7770409f9c194e\": rpc error: code = NotFound desc = could not find container \"20d35ec6d07ff7fb4194128f42ebbc5cc3bce3071306d400ab7770409f9c194e\": container with ID starting with 20d35ec6d07ff7fb4194128f42ebbc5cc3bce3071306d400ab7770409f9c194e not found: ID does not exist" Nov 03 21:07:09.053652 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:09.053621 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5e4bf14-js862"] Nov 03 21:07:09.103466 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:09.103378 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e971f2a5-w98b8"] Nov 03 21:07:09.108343 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:09.108314 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5e4bf14-js862"] Nov 03 21:07:09.113343 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:09.113317 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e971f2a5-w98b8"] Nov 03 21:07:09.118633 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:09.118607 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e3917193-665777fcf9-khvcd"] Nov 03 21:07:09.152843 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:09.152780 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e3917193-665777fcf9-khvcd"] Nov 03 21:07:09.259852 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:09.259817 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e3917193-68c654f4dd-ttdqc"] Nov 03 21:07:09.264738 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:09.264704 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e3917193-68c654f4dd-ttdqc"] Nov 03 21:07:09.955051 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:09.955018 4305 generic.go:356] "Generic (PLEG): container finished" podID="d35870fa-5e32-4297-ba1b-4f6526cf4bfd" containerID="74828063f3048ec2b674488e221b95d6107efb955c64ef034de43bef4eb62b9b" exitCode=0 Nov 03 21:07:09.955561 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:09.955054 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk" event={"ID":"d35870fa-5e32-4297-ba1b-4f6526cf4bfd","Type":"ContainerDied","Data":"74828063f3048ec2b674488e221b95d6107efb955c64ef034de43bef4eb62b9b"} Nov 03 21:07:09.955640 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:09.955626 4305 scope.go:122] "RemoveContainer" containerID="74828063f3048ec2b674488e221b95d6107efb955c64ef034de43bef4eb62b9b" Nov 03 21:07:09.957820 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:09.957788 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:07:09.957954 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:09.957876 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:07:09.957954 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:09.957929 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:07:10.897143 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:10.897112 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="261e3cbf-4f0e-4fc4-9e2f-11dcd6298ee5" path="/var/lib/kubelet/pods/261e3cbf-4f0e-4fc4-9e2f-11dcd6298ee5/volumes" Nov 03 21:07:10.897654 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:10.897641 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="515e30b7-016e-4bbf-b546-66900d5f5e3d" path="/var/lib/kubelet/pods/515e30b7-016e-4bbf-b546-66900d5f5e3d/volumes" Nov 03 21:07:10.898035 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:10.898022 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="998a67c4-3f64-4922-a2c6-4b7629c96b28" path="/var/lib/kubelet/pods/998a67c4-3f64-4922-a2c6-4b7629c96b28/volumes" Nov 03 21:07:10.898564 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:10.898541 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="facfcb94-b6b7-4072-b4f0-49ef05e5dde8" path="/var/lib/kubelet/pods/facfcb94-b6b7-4072-b4f0-49ef05e5dde8/volumes" Nov 03 21:07:10.978053 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:10.978006 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk" event={"ID":"d35870fa-5e32-4297-ba1b-4f6526cf4bfd","Type":"ContainerStarted","Data":"8acdbae0a25b9852b174ac8f53290dbcb0d7c8e980d968f171d0eed79d380641"} Nov 03 21:07:11.292559 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:11.292428 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk" Nov 03 21:07:11.293681 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:11.293651 4305 patch_prober.go:29] interesting pod/e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Nov 03 21:07:11.293833 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:11.293715 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk" podUID="d35870fa-5e32-4297-ba1b-4f6526cf4bfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Nov 03 21:07:12.292799 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:12.292762 4305 patch_prober.go:29] interesting pod/e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Nov 03 21:07:12.293239 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:12.292827 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk" podUID="d35870fa-5e32-4297-ba1b-4f6526cf4bfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Nov 03 21:07:12.990687 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:12.990655 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk" Nov 03 21:07:13.292874 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:13.292797 4305 patch_prober.go:29] interesting pod/e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Nov 03 21:07:13.292874 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:13.292862 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk" podUID="d35870fa-5e32-4297-ba1b-4f6526cf4bfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Nov 03 21:07:14.293145 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:14.293107 4305 patch_prober.go:29] interesting pod/e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Nov 03 21:07:14.293581 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:14.293193 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk" podUID="d35870fa-5e32-4297-ba1b-4f6526cf4bfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Nov 03 21:07:15.292540 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:15.292496 4305 patch_prober.go:29] interesting pod/e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Nov 03 21:07:15.292753 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:15.292562 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk" podUID="d35870fa-5e32-4297-ba1b-4f6526cf4bfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Nov 03 21:07:16.293105 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:16.293067 4305 patch_prober.go:29] interesting pod/e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Nov 03 21:07:16.293635 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:16.293131 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk" podUID="d35870fa-5e32-4297-ba1b-4f6526cf4bfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Nov 03 21:07:16.325187 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:07:16.325147 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:07:17.298196 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:17.298156 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk" Nov 03 21:07:17.299500 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:17.299426 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk" Nov 03 21:07:19.069891 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:19.069858 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c1d3bc7-mfc85"] Nov 03 21:07:19.119234 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:19.119204 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c1d3bc7-mfc85"] Nov 03 21:07:20.085462 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:20.084834 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0af8368d-btdft"] Nov 03 21:07:20.126819 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:20.126778 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0af8368d-btdft"] Nov 03 21:07:20.904421 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:20.904386 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b8cdfb4-1311-4d1c-8b40-64ce920f0248" path="/var/lib/kubelet/pods/2b8cdfb4-1311-4d1c-8b40-64ce920f0248/volumes" Nov 03 21:07:20.905155 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:20.905137 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3fa728b-bd28-42b6-98de-9853c0a04f15" path="/var/lib/kubelet/pods/d3fa728b-bd28-42b6-98de-9853c0a04f15/volumes" Nov 03 21:07:30.787388 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:30.787355 4305 scope.go:122] "RemoveContainer" containerID="429c663c84c04c97f21a4985e25896f2bd1437d74c43056a7f07400f67da94e7" Nov 03 21:07:30.802650 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:30.802621 4305 scope.go:122] "RemoveContainer" containerID="f6f1b0e91566d64933a43114d277e1241c02a15e94cb0acac95e4c1bc6839a24" Nov 03 21:07:30.813488 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:30.813462 4305 scope.go:122] "RemoveContainer" containerID="d2b0f0ed4c2715033c8d86fa3586aedabf80d00b1ed36b12ecedd9556356aeeb" Nov 03 21:07:30.824573 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:30.824549 4305 scope.go:122] "RemoveContainer" containerID="5a7e9a46ee4461e374db17845ca5c132fa94c0b74d53ea9f6345da8b67c08b44" Nov 03 21:07:30.839513 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:30.839484 4305 scope.go:122] "RemoveContainer" containerID="614f822ce753f32931aff59de48a464fabcd99f3661ebe3d36a4860cd3f389c7" Nov 03 21:07:30.850367 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:30.850349 4305 scope.go:122] "RemoveContainer" containerID="b92cbc038d5b3ce38786fc7f9871117629f4454fde3c707dfc65d4f4a6386004" Nov 03 21:07:30.860816 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:30.860791 4305 scope.go:122] "RemoveContainer" containerID="ce40284ca2fc4ee9e0a14a312855808de1f3988f4e0e213c0ab74e921fd420f8" Nov 03 21:07:30.871120 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:30.871101 4305 scope.go:122] "RemoveContainer" containerID="982701db3b1034b59f2c8924e4608dfa84a079f8e999739fe329f0ec42ea2b27" Nov 03 21:07:31.056966 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:31.056820 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-661f63e8-h52c9"] Nov 03 21:07:31.103687 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:31.103652 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-661f63e8-h52c9"] Nov 03 21:07:32.897690 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:32.897651 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a81f4383-a20c-4c43-824a-2987ff0c8f38" path="/var/lib/kubelet/pods/a81f4383-a20c-4c43-824a-2987ff0c8f38/volumes" Nov 03 21:07:33.188598 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:33.188513 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e41159db-6l7cz"] Nov 03 21:07:33.189690 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:33.189671 4305 cpu_manager.go:414] "containerMap: removing container" podUID="998a67c4-3f64-4922-a2c6-4b7629c96b28" containerName="payload-pre-processing" Nov 03 21:07:33.189748 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:33.189694 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="998a67c4-3f64-4922-a2c6-4b7629c96b28" containerName="payload-pre-processing" Nov 03 21:07:33.189748 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:33.189732 4305 cpu_manager.go:414] "containerMap: removing container" podUID="facfcb94-b6b7-4072-b4f0-49ef05e5dde8" containerName="payload-processing" Nov 03 21:07:33.189748 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:33.189740 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="facfcb94-b6b7-4072-b4f0-49ef05e5dde8" containerName="payload-processing" Nov 03 21:07:33.189844 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:33.189752 4305 cpu_manager.go:414] "containerMap: removing container" podUID="a170fb6b-d3eb-474a-a4cc-1fced5daf1c5" containerName="revoke-keys" Nov 03 21:07:33.189844 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:33.189759 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a170fb6b-d3eb-474a-a4cc-1fced5daf1c5" containerName="revoke-keys" Nov 03 21:07:33.189980 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:33.189968 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="facfcb94-b6b7-4072-b4f0-49ef05e5dde8" containerName="payload-processing" Nov 03 21:07:33.190025 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:33.189985 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="a170fb6b-d3eb-474a-a4cc-1fced5daf1c5" containerName="revoke-keys" Nov 03 21:07:33.190025 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:33.190009 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="998a67c4-3f64-4922-a2c6-4b7629c96b28" containerName="payload-pre-processing" Nov 03 21:07:33.196635 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:33.196614 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e41159db-6l7cz" Nov 03 21:07:33.249166 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:33.249138 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:07:33.249707 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:33.249683 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e41159db-6l7cz"] Nov 03 21:07:33.292936 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:33.292910 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77vtd\"" Nov 03 21:07:33.293159 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:33.293031 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14573b6a-c19e-42fe-85c6-f7ffa46db476-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e41159db-6l7cz\" (UID: \"14573b6a-c19e-42fe-85c6-f7ffa46db476\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e41159db-6l7cz" Nov 03 21:07:33.293159 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:33.293080 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8btr\" (UniqueName: \"kubernetes.io/projected/14573b6a-c19e-42fe-85c6-f7ffa46db476-kube-api-access-w8btr\") pod \"maas-api-revoke-keys-e2e-ait-e41159db-6l7cz\" (UID: \"14573b6a-c19e-42fe-85c6-f7ffa46db476\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e41159db-6l7cz" Nov 03 21:07:33.293522 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:33.293218 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:07:33.293522 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:33.293302 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:07:33.293522 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:33.293333 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14573b6a-c19e-42fe-85c6-f7ffa46db476-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e41159db-6l7cz\" (UID: \"14573b6a-c19e-42fe-85c6-f7ffa46db476\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e41159db-6l7cz" Nov 03 21:07:33.395551 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:33.395519 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14573b6a-c19e-42fe-85c6-f7ffa46db476-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e41159db-6l7cz\" (UID: \"14573b6a-c19e-42fe-85c6-f7ffa46db476\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e41159db-6l7cz" Nov 03 21:07:33.395551 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:33.395562 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8btr\" (UniqueName: \"kubernetes.io/projected/14573b6a-c19e-42fe-85c6-f7ffa46db476-kube-api-access-w8btr\") pod \"maas-api-revoke-keys-e2e-ait-e41159db-6l7cz\" (UID: \"14573b6a-c19e-42fe-85c6-f7ffa46db476\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e41159db-6l7cz" Nov 03 21:07:33.395826 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:33.395684 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14573b6a-c19e-42fe-85c6-f7ffa46db476-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e41159db-6l7cz\" (UID: \"14573b6a-c19e-42fe-85c6-f7ffa46db476\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e41159db-6l7cz" Nov 03 21:07:33.396243 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:33.396218 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14573b6a-c19e-42fe-85c6-f7ffa46db476-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e41159db-6l7cz\" (UID: \"14573b6a-c19e-42fe-85c6-f7ffa46db476\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e41159db-6l7cz" Nov 03 21:07:33.398227 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:33.398203 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14573b6a-c19e-42fe-85c6-f7ffa46db476-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e41159db-6l7cz\" (UID: \"14573b6a-c19e-42fe-85c6-f7ffa46db476\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e41159db-6l7cz" Nov 03 21:07:33.459872 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:33.459787 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8btr\" (UniqueName: \"kubernetes.io/projected/14573b6a-c19e-42fe-85c6-f7ffa46db476-kube-api-access-w8btr\") pod \"maas-api-revoke-keys-e2e-ait-e41159db-6l7cz\" (UID: \"14573b6a-c19e-42fe-85c6-f7ffa46db476\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e41159db-6l7cz" Nov 03 21:07:33.512557 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:33.512414 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e41159db-6l7cz" Nov 03 21:07:33.613221 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:33.613189 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04f5216c-4shmx"] Nov 03 21:07:33.619104 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:33.619074 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04f5216c-4shmx" Nov 03 21:07:33.682740 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:33.682701 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04f5216c-4shmx"] Nov 03 21:07:33.700789 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:33.700749 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzxcb\" (UniqueName: \"kubernetes.io/projected/075cc1fc-27d6-42a9-a068-1240584106bc-kube-api-access-dzxcb\") pod \"maas-api-revoke-keys-e2e-ait-04f5216c-4shmx\" (UID: \"075cc1fc-27d6-42a9-a068-1240584106bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04f5216c-4shmx" Nov 03 21:07:33.700992 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:33.700960 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/075cc1fc-27d6-42a9-a068-1240584106bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-04f5216c-4shmx\" (UID: \"075cc1fc-27d6-42a9-a068-1240584106bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04f5216c-4shmx" Nov 03 21:07:33.701251 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:33.701231 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/075cc1fc-27d6-42a9-a068-1240584106bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-04f5216c-4shmx\" (UID: \"075cc1fc-27d6-42a9-a068-1240584106bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04f5216c-4shmx" Nov 03 21:07:33.802623 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:33.802520 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/075cc1fc-27d6-42a9-a068-1240584106bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-04f5216c-4shmx\" (UID: \"075cc1fc-27d6-42a9-a068-1240584106bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04f5216c-4shmx" Nov 03 21:07:33.802813 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:33.802761 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/075cc1fc-27d6-42a9-a068-1240584106bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-04f5216c-4shmx\" (UID: \"075cc1fc-27d6-42a9-a068-1240584106bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04f5216c-4shmx" Nov 03 21:07:33.802892 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:33.802829 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzxcb\" (UniqueName: \"kubernetes.io/projected/075cc1fc-27d6-42a9-a068-1240584106bc-kube-api-access-dzxcb\") pod \"maas-api-revoke-keys-e2e-ait-04f5216c-4shmx\" (UID: \"075cc1fc-27d6-42a9-a068-1240584106bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04f5216c-4shmx" Nov 03 21:07:33.803158 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:33.803137 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/075cc1fc-27d6-42a9-a068-1240584106bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-04f5216c-4shmx\" (UID: \"075cc1fc-27d6-42a9-a068-1240584106bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04f5216c-4shmx" Nov 03 21:07:33.805322 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:33.805302 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/075cc1fc-27d6-42a9-a068-1240584106bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-04f5216c-4shmx\" (UID: \"075cc1fc-27d6-42a9-a068-1240584106bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04f5216c-4shmx" Nov 03 21:07:33.864075 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:33.864044 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzxcb\" (UniqueName: \"kubernetes.io/projected/075cc1fc-27d6-42a9-a068-1240584106bc-kube-api-access-dzxcb\") pod \"maas-api-revoke-keys-e2e-ait-04f5216c-4shmx\" (UID: \"075cc1fc-27d6-42a9-a068-1240584106bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04f5216c-4shmx" Nov 03 21:07:33.901645 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:07:33.901615 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14573b6a_c19e_42fe_85c6_f7ffa46db476.slice/crio-83e88bdebb03014cd6d258fa9aae164838196fd66daad08c5cab7f1b0edcd1c6 WatchSource:0}: Error finding container 83e88bdebb03014cd6d258fa9aae164838196fd66daad08c5cab7f1b0edcd1c6: Status 404 returned error can't find the container with id 83e88bdebb03014cd6d258fa9aae164838196fd66daad08c5cab7f1b0edcd1c6 Nov 03 21:07:33.902079 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:33.901706 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e41159db-6l7cz"] Nov 03 21:07:33.936031 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:33.936007 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04f5216c-4shmx" Nov 03 21:07:34.200015 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:34.199977 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e41159db-6l7cz" event={"ID":"14573b6a-c19e-42fe-85c6-f7ffa46db476","Type":"ContainerStarted","Data":"83e88bdebb03014cd6d258fa9aae164838196fd66daad08c5cab7f1b0edcd1c6"} Nov 03 21:07:34.320366 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:07:34.320333 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod075cc1fc_27d6_42a9_a068_1240584106bc.slice/crio-ba43d93d5e77b9b24b039fa42b799c0ebca47ea9eac9596d12d082e7c1d4068a WatchSource:0}: Error finding container ba43d93d5e77b9b24b039fa42b799c0ebca47ea9eac9596d12d082e7c1d4068a: Status 404 returned error can't find the container with id ba43d93d5e77b9b24b039fa42b799c0ebca47ea9eac9596d12d082e7c1d4068a Nov 03 21:07:34.320799 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:34.320773 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04f5216c-4shmx"] Nov 03 21:07:35.084016 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:35.083979 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cf9b10d-ktw77"] Nov 03 21:07:35.088332 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:35.088307 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cf9b10d-ktw77"] Nov 03 21:07:35.212308 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:35.212275 4305 generic.go:356] "Generic (PLEG): container finished" podID="14573b6a-c19e-42fe-85c6-f7ffa46db476" containerID="8694ce683875e96acaa705b446df26bf253f5aec162f7500538b0c74f0fffddb" exitCode=0 Nov 03 21:07:35.212540 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:35.212368 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e41159db-6l7cz" event={"ID":"14573b6a-c19e-42fe-85c6-f7ffa46db476","Type":"ContainerDied","Data":"8694ce683875e96acaa705b446df26bf253f5aec162f7500538b0c74f0fffddb"} Nov 03 21:07:35.214174 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:35.214142 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04f5216c-4shmx" event={"ID":"075cc1fc-27d6-42a9-a068-1240584106bc","Type":"ContainerStarted","Data":"8c45ca20ee60a442b554da2588707fbf2c60bc1b19a26aab9296279b1cf37ea5"} Nov 03 21:07:35.214298 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:35.214197 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04f5216c-4shmx" event={"ID":"075cc1fc-27d6-42a9-a068-1240584106bc","Type":"ContainerStarted","Data":"ba43d93d5e77b9b24b039fa42b799c0ebca47ea9eac9596d12d082e7c1d4068a"} Nov 03 21:07:35.443709 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:35.443649 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04f5216c-4shmx" podStartSLOduration=2.443633086 podStartE2EDuration="2.443633086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:07:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:07:35.440660091 +0000 UTC m=+1690.665979883" watchObservedRunningTime="2026-11-03 21:07:35.443633086 +0000 UTC m=+1690.668952877" Nov 03 21:07:36.229875 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:36.229836 4305 generic.go:356] "Generic (PLEG): container finished" podID="075cc1fc-27d6-42a9-a068-1240584106bc" containerID="8c45ca20ee60a442b554da2588707fbf2c60bc1b19a26aab9296279b1cf37ea5" exitCode=0 Nov 03 21:07:36.230602 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:36.229888 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04f5216c-4shmx" event={"ID":"075cc1fc-27d6-42a9-a068-1240584106bc","Type":"ContainerDied","Data":"8c45ca20ee60a442b554da2588707fbf2c60bc1b19a26aab9296279b1cf37ea5"} Nov 03 21:07:36.496806 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:36.496776 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e41159db-6l7cz" Nov 03 21:07:36.662570 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:36.662525 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14573b6a-c19e-42fe-85c6-f7ffa46db476-kube-api-access-w8btr\" (UniqueName: \"kubernetes.io/projected/14573b6a-c19e-42fe-85c6-f7ffa46db476-kube-api-access-w8btr\") pod \"14573b6a-c19e-42fe-85c6-f7ffa46db476\" (UID: \"14573b6a-c19e-42fe-85c6-f7ffa46db476\") " Nov 03 21:07:36.662755 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:36.662706 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14573b6a-c19e-42fe-85c6-f7ffa46db476-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14573b6a-c19e-42fe-85c6-f7ffa46db476-maas-api-xks-ca\") pod \"14573b6a-c19e-42fe-85c6-f7ffa46db476\" (UID: \"14573b6a-c19e-42fe-85c6-f7ffa46db476\") " Nov 03 21:07:36.662872 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:36.662854 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14573b6a-c19e-42fe-85c6-f7ffa46db476-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14573b6a-c19e-42fe-85c6-f7ffa46db476-maas-api-openshift-ca\") pod \"14573b6a-c19e-42fe-85c6-f7ffa46db476\" (UID: \"14573b6a-c19e-42fe-85c6-f7ffa46db476\") " Nov 03 21:07:36.663513 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:36.663479 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14573b6a-c19e-42fe-85c6-f7ffa46db476-maas-api-openshift-ca" pod "14573b6a-c19e-42fe-85c6-f7ffa46db476" (UID: "14573b6a-c19e-42fe-85c6-f7ffa46db476"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:07:36.664968 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:36.664939 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14573b6a-c19e-42fe-85c6-f7ffa46db476-maas-api-xks-ca" pod "14573b6a-c19e-42fe-85c6-f7ffa46db476" (UID: "14573b6a-c19e-42fe-85c6-f7ffa46db476"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:07:36.665078 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:36.664990 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14573b6a-c19e-42fe-85c6-f7ffa46db476-kube-api-access-w8btr" pod "14573b6a-c19e-42fe-85c6-f7ffa46db476" (UID: "14573b6a-c19e-42fe-85c6-f7ffa46db476"). InnerVolumeSpecName "kube-api-access-w8btr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:07:36.765580 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:36.765499 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14573b6a-c19e-42fe-85c6-f7ffa46db476-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:07:36.765580 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:36.765528 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14573b6a-c19e-42fe-85c6-f7ffa46db476-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:07:36.765580 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:36.765538 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8btr\" (UniqueName: \"kubernetes.io/projected/14573b6a-c19e-42fe-85c6-f7ffa46db476-kube-api-access-w8btr\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:07:36.900992 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:36.900960 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fa000fe-eafe-4adb-a4a7-28251831502d" path="/var/lib/kubelet/pods/2fa000fe-eafe-4adb-a4a7-28251831502d/volumes" Nov 03 21:07:37.240589 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:37.240558 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e41159db-6l7cz" Nov 03 21:07:37.241077 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:37.240600 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e41159db-6l7cz" event={"ID":"14573b6a-c19e-42fe-85c6-f7ffa46db476","Type":"ContainerDied","Data":"83e88bdebb03014cd6d258fa9aae164838196fd66daad08c5cab7f1b0edcd1c6"} Nov 03 21:07:37.241077 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:37.240645 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83e88bdebb03014cd6d258fa9aae164838196fd66daad08c5cab7f1b0edcd1c6" Nov 03 21:07:37.420901 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:37.420874 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04f5216c-4shmx" Nov 03 21:07:37.577692 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:37.577604 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/075cc1fc-27d6-42a9-a068-1240584106bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/075cc1fc-27d6-42a9-a068-1240584106bc-maas-api-xks-ca\") pod \"075cc1fc-27d6-42a9-a068-1240584106bc\" (UID: \"075cc1fc-27d6-42a9-a068-1240584106bc\") " Nov 03 21:07:37.577692 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:37.577656 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/075cc1fc-27d6-42a9-a068-1240584106bc-kube-api-access-dzxcb\" (UniqueName: \"kubernetes.io/projected/075cc1fc-27d6-42a9-a068-1240584106bc-kube-api-access-dzxcb\") pod \"075cc1fc-27d6-42a9-a068-1240584106bc\" (UID: \"075cc1fc-27d6-42a9-a068-1240584106bc\") " Nov 03 21:07:37.577935 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:37.577874 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/075cc1fc-27d6-42a9-a068-1240584106bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/075cc1fc-27d6-42a9-a068-1240584106bc-maas-api-openshift-ca\") pod \"075cc1fc-27d6-42a9-a068-1240584106bc\" (UID: \"075cc1fc-27d6-42a9-a068-1240584106bc\") " Nov 03 21:07:37.578297 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:37.578249 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/075cc1fc-27d6-42a9-a068-1240584106bc-maas-api-openshift-ca" pod "075cc1fc-27d6-42a9-a068-1240584106bc" (UID: "075cc1fc-27d6-42a9-a068-1240584106bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:07:37.579578 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:37.579553 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/075cc1fc-27d6-42a9-a068-1240584106bc-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:07:37.580349 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:37.580281 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/075cc1fc-27d6-42a9-a068-1240584106bc-maas-api-xks-ca" pod "075cc1fc-27d6-42a9-a068-1240584106bc" (UID: "075cc1fc-27d6-42a9-a068-1240584106bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:07:37.582037 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:37.582007 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/075cc1fc-27d6-42a9-a068-1240584106bc-kube-api-access-dzxcb" pod "075cc1fc-27d6-42a9-a068-1240584106bc" (UID: "075cc1fc-27d6-42a9-a068-1240584106bc"). InnerVolumeSpecName "kube-api-access-dzxcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:07:37.681011 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:37.680972 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/075cc1fc-27d6-42a9-a068-1240584106bc-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:07:37.681011 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:37.681007 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzxcb\" (UniqueName: \"kubernetes.io/projected/075cc1fc-27d6-42a9-a068-1240584106bc-kube-api-access-dzxcb\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:07:38.050700 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:38.050657 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fd23f4f-5759r"] Nov 03 21:07:38.054951 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:38.054926 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fd23f4f-5759r"] Nov 03 21:07:38.249942 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:38.249913 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04f5216c-4shmx" Nov 03 21:07:38.250332 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:38.249946 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04f5216c-4shmx" event={"ID":"075cc1fc-27d6-42a9-a068-1240584106bc","Type":"ContainerDied","Data":"ba43d93d5e77b9b24b039fa42b799c0ebca47ea9eac9596d12d082e7c1d4068a"} Nov 03 21:07:38.250332 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:38.249998 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba43d93d5e77b9b24b039fa42b799c0ebca47ea9eac9596d12d082e7c1d4068a" Nov 03 21:07:38.897011 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:38.896973 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="878a0158-63b8-43da-8925-bb196f38da4f" path="/var/lib/kubelet/pods/878a0158-63b8-43da-8925-bb196f38da4f/volumes" Nov 03 21:07:39.737813 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:39.737776 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v"] Nov 03 21:07:39.738329 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:39.738075 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v" podUID="26613665-1067-4ba1-9b16-f84e8c583c2a" containerName="istio-proxy" containerID="cri-o://bdd21d6639c9bd024bc58a627e4e6f119dda274b402f4315f27dc4baf0d8eb52" gracePeriod=30 Nov 03 21:07:43.578006 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:43.577965 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e41159db-6967b48544-cwjl7"] Nov 03 21:07:45.034270 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:45.034211 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v" Nov 03 21:07:45.130917 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:45.130071 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26613665-1067-4ba1-9b16-f84e8c583c2a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26613665-1067-4ba1-9b16-f84e8c583c2a-workload-socket\") pod \"26613665-1067-4ba1-9b16-f84e8c583c2a\" (UID: \"26613665-1067-4ba1-9b16-f84e8c583c2a\") " Nov 03 21:07:45.130917 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:45.130145 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26613665-1067-4ba1-9b16-f84e8c583c2a-kube-api-access-4t2qs\" (UniqueName: \"kubernetes.io/projected/26613665-1067-4ba1-9b16-f84e8c583c2a-kube-api-access-4t2qs\") pod \"26613665-1067-4ba1-9b16-f84e8c583c2a\" (UID: \"26613665-1067-4ba1-9b16-f84e8c583c2a\") " Nov 03 21:07:45.130917 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:45.130256 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26613665-1067-4ba1-9b16-f84e8c583c2a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26613665-1067-4ba1-9b16-f84e8c583c2a-workload-certs\") pod \"26613665-1067-4ba1-9b16-f84e8c583c2a\" (UID: \"26613665-1067-4ba1-9b16-f84e8c583c2a\") " Nov 03 21:07:45.130917 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:45.130350 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26613665-1067-4ba1-9b16-f84e8c583c2a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26613665-1067-4ba1-9b16-f84e8c583c2a-credential-socket\") pod \"26613665-1067-4ba1-9b16-f84e8c583c2a\" (UID: \"26613665-1067-4ba1-9b16-f84e8c583c2a\") " Nov 03 21:07:45.130917 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:45.130451 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26613665-1067-4ba1-9b16-f84e8c583c2a-istio-token\" (UniqueName: \"kubernetes.io/projected/26613665-1067-4ba1-9b16-f84e8c583c2a-istio-token\") pod \"26613665-1067-4ba1-9b16-f84e8c583c2a\" (UID: \"26613665-1067-4ba1-9b16-f84e8c583c2a\") " Nov 03 21:07:45.130917 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:45.130511 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26613665-1067-4ba1-9b16-f84e8c583c2a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26613665-1067-4ba1-9b16-f84e8c583c2a-istio-data\") pod \"26613665-1067-4ba1-9b16-f84e8c583c2a\" (UID: \"26613665-1067-4ba1-9b16-f84e8c583c2a\") " Nov 03 21:07:45.130917 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:45.130607 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26613665-1067-4ba1-9b16-f84e8c583c2a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26613665-1067-4ba1-9b16-f84e8c583c2a-istio-envoy\") pod \"26613665-1067-4ba1-9b16-f84e8c583c2a\" (UID: \"26613665-1067-4ba1-9b16-f84e8c583c2a\") " Nov 03 21:07:45.130917 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:45.130757 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/26613665-1067-4ba1-9b16-f84e8c583c2a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26613665-1067-4ba1-9b16-f84e8c583c2a-istio-podinfo\") pod \"26613665-1067-4ba1-9b16-f84e8c583c2a\" (UID: \"26613665-1067-4ba1-9b16-f84e8c583c2a\") " Nov 03 21:07:45.130917 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:45.130792 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26613665-1067-4ba1-9b16-f84e8c583c2a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26613665-1067-4ba1-9b16-f84e8c583c2a-istiod-ca-cert\") pod \"26613665-1067-4ba1-9b16-f84e8c583c2a\" (UID: \"26613665-1067-4ba1-9b16-f84e8c583c2a\") " Nov 03 21:07:45.132810 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:45.131272 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26613665-1067-4ba1-9b16-f84e8c583c2a-istiod-ca-cert" pod "26613665-1067-4ba1-9b16-f84e8c583c2a" (UID: "26613665-1067-4ba1-9b16-f84e8c583c2a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:07:45.132810 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:45.131616 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26613665-1067-4ba1-9b16-f84e8c583c2a-workload-socket" pod "26613665-1067-4ba1-9b16-f84e8c583c2a" (UID: "26613665-1067-4ba1-9b16-f84e8c583c2a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:07:45.132810 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:45.132770 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26613665-1067-4ba1-9b16-f84e8c583c2a-istiod-ca-cert\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:07:45.132810 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:45.132793 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26613665-1067-4ba1-9b16-f84e8c583c2a-workload-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:07:45.150459 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:45.133627 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26613665-1067-4ba1-9b16-f84e8c583c2a-workload-certs" pod "26613665-1067-4ba1-9b16-f84e8c583c2a" (UID: "26613665-1067-4ba1-9b16-f84e8c583c2a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:07:45.150459 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:45.133810 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26613665-1067-4ba1-9b16-f84e8c583c2a-credential-socket" pod "26613665-1067-4ba1-9b16-f84e8c583c2a" (UID: "26613665-1067-4ba1-9b16-f84e8c583c2a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:07:45.150459 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:45.134716 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26613665-1067-4ba1-9b16-f84e8c583c2a-istio-data" pod "26613665-1067-4ba1-9b16-f84e8c583c2a" (UID: "26613665-1067-4ba1-9b16-f84e8c583c2a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:07:45.189474 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:45.189129 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26613665-1067-4ba1-9b16-f84e8c583c2a-istio-token" pod "26613665-1067-4ba1-9b16-f84e8c583c2a" (UID: "26613665-1067-4ba1-9b16-f84e8c583c2a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:07:45.190293 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:45.190258 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26613665-1067-4ba1-9b16-f84e8c583c2a-kube-api-access-4t2qs" pod "26613665-1067-4ba1-9b16-f84e8c583c2a" (UID: "26613665-1067-4ba1-9b16-f84e8c583c2a"). InnerVolumeSpecName "kube-api-access-4t2qs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:07:45.191631 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:45.191590 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26613665-1067-4ba1-9b16-f84e8c583c2a-istio-envoy" pod "26613665-1067-4ba1-9b16-f84e8c583c2a" (UID: "26613665-1067-4ba1-9b16-f84e8c583c2a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:07:45.193479 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:45.192229 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26613665-1067-4ba1-9b16-f84e8c583c2a-istio-podinfo" pod "26613665-1067-4ba1-9b16-f84e8c583c2a" (UID: "26613665-1067-4ba1-9b16-f84e8c583c2a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:07:45.245721 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:45.240173 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t2qs\" (UniqueName: \"kubernetes.io/projected/26613665-1067-4ba1-9b16-f84e8c583c2a-kube-api-access-4t2qs\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:07:45.245721 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:45.240210 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26613665-1067-4ba1-9b16-f84e8c583c2a-workload-certs\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:07:45.245721 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:45.240226 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26613665-1067-4ba1-9b16-f84e8c583c2a-credential-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:07:45.245721 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:45.240239 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26613665-1067-4ba1-9b16-f84e8c583c2a-istio-token\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:07:45.245721 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:45.240254 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26613665-1067-4ba1-9b16-f84e8c583c2a-istio-data\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:07:45.245721 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:45.240267 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26613665-1067-4ba1-9b16-f84e8c583c2a-istio-envoy\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:07:45.245721 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:45.240281 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26613665-1067-4ba1-9b16-f84e8c583c2a-istio-podinfo\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:07:45.397296 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:45.397049 4305 generic.go:356] "Generic (PLEG): container finished" podID="26613665-1067-4ba1-9b16-f84e8c583c2a" containerID="bdd21d6639c9bd024bc58a627e4e6f119dda274b402f4315f27dc4baf0d8eb52" exitCode=0 Nov 03 21:07:45.397296 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:45.397150 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v" event={"ID":"26613665-1067-4ba1-9b16-f84e8c583c2a","Type":"ContainerDied","Data":"bdd21d6639c9bd024bc58a627e4e6f119dda274b402f4315f27dc4baf0d8eb52"} Nov 03 21:07:45.397296 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:45.397198 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v" event={"ID":"26613665-1067-4ba1-9b16-f84e8c583c2a","Type":"ContainerDied","Data":"7a8de5f3446cbc61d84b9953de793ee6907da868f57f6cf4d763c54b45239cd1"} Nov 03 21:07:45.397296 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:45.397227 4305 scope.go:122] "RemoveContainer" containerID="bdd21d6639c9bd024bc58a627e4e6f119dda274b402f4315f27dc4baf0d8eb52" Nov 03 21:07:45.400288 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:45.397630 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v" Nov 03 21:07:45.414798 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:45.414773 4305 scope.go:122] "RemoveContainer" containerID="bdd21d6639c9bd024bc58a627e4e6f119dda274b402f4315f27dc4baf0d8eb52" Nov 03 21:07:45.415164 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:07:45.415138 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdd21d6639c9bd024bc58a627e4e6f119dda274b402f4315f27dc4baf0d8eb52\": container with ID starting with bdd21d6639c9bd024bc58a627e4e6f119dda274b402f4315f27dc4baf0d8eb52 not found: ID does not exist" containerID="bdd21d6639c9bd024bc58a627e4e6f119dda274b402f4315f27dc4baf0d8eb52" Nov 03 21:07:45.415246 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:45.415177 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdd21d6639c9bd024bc58a627e4e6f119dda274b402f4315f27dc4baf0d8eb52"} err="failed to get container status \"bdd21d6639c9bd024bc58a627e4e6f119dda274b402f4315f27dc4baf0d8eb52\": rpc error: code = NotFound desc = could not find container \"bdd21d6639c9bd024bc58a627e4e6f119dda274b402f4315f27dc4baf0d8eb52\": container with ID starting with bdd21d6639c9bd024bc58a627e4e6f119dda274b402f4315f27dc4baf0d8eb52 not found: ID does not exist" Nov 03 21:07:45.596771 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:45.596737 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v"] Nov 03 21:07:45.603893 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:45.603859 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-69v4v"] Nov 03 21:07:46.148816 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:46.148782 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-04f5216c-b64d9fdfb-p8dxz"] Nov 03 21:07:46.248175 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:46.248142 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-04f5216c-fb66cddc-vbsb7"] Nov 03 21:07:46.325118 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:07:46.325076 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:07:46.899600 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:46.899565 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26613665-1067-4ba1-9b16-f84e8c583c2a" path="/var/lib/kubelet/pods/26613665-1067-4ba1-9b16-f84e8c583c2a/volumes" Nov 03 21:07:48.578617 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:48.578569 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-e41159db-6967b48544-cwjl7" podUID="cd55d060-ec86-413c-8303-1d7bde266156" containerName="payload-processing" containerID="cri-o://8f55dcd9b33bd2860bc96e9dac6a1e017a8daeea77c363819a5e1ac956f85d24" gracePeriod=25 Nov 03 21:07:48.894533 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:48.894502 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e41159db-6967b48544-cwjl7" Nov 03 21:07:48.920416 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:48.920383 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0236317-jcldz"] Nov 03 21:07:48.921201 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:48.921180 4305 cpu_manager.go:414] "containerMap: removing container" podUID="cd55d060-ec86-413c-8303-1d7bde266156" containerName="payload-processing" Nov 03 21:07:48.921201 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:48.921201 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd55d060-ec86-413c-8303-1d7bde266156" containerName="payload-processing" Nov 03 21:07:48.921365 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:48.921220 4305 cpu_manager.go:414] "containerMap: removing container" podUID="26613665-1067-4ba1-9b16-f84e8c583c2a" containerName="istio-proxy" Nov 03 21:07:48.921365 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:48.921227 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26613665-1067-4ba1-9b16-f84e8c583c2a" containerName="istio-proxy" Nov 03 21:07:48.921365 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:48.921267 4305 cpu_manager.go:414] "containerMap: removing container" podUID="14573b6a-c19e-42fe-85c6-f7ffa46db476" containerName="revoke-keys" Nov 03 21:07:48.921365 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:48.921275 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14573b6a-c19e-42fe-85c6-f7ffa46db476" containerName="revoke-keys" Nov 03 21:07:48.921365 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:48.921305 4305 cpu_manager.go:414] "containerMap: removing container" podUID="075cc1fc-27d6-42a9-a068-1240584106bc" containerName="revoke-keys" Nov 03 21:07:48.921365 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:48.921312 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="075cc1fc-27d6-42a9-a068-1240584106bc" containerName="revoke-keys" Nov 03 21:07:48.921781 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:48.921525 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="14573b6a-c19e-42fe-85c6-f7ffa46db476" containerName="revoke-keys" Nov 03 21:07:48.921781 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:48.921546 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="075cc1fc-27d6-42a9-a068-1240584106bc" containerName="revoke-keys" Nov 03 21:07:48.921781 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:48.921566 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd55d060-ec86-413c-8303-1d7bde266156" containerName="payload-processing" Nov 03 21:07:48.921781 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:48.921575 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="26613665-1067-4ba1-9b16-f84e8c583c2a" containerName="istio-proxy" Nov 03 21:07:48.925318 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:48.925298 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0236317-jcldz" Nov 03 21:07:48.977645 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:48.977617 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:07:48.983080 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:48.983053 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0236317-jcldz"] Nov 03 21:07:48.996920 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:48.996898 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:07:48.997150 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:48.997127 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77vtd\"" Nov 03 21:07:48.997215 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:48.997193 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:07:49.044966 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:49.044924 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd55d060-ec86-413c-8303-1d7bde266156-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd55d060-ec86-413c-8303-1d7bde266156-plugins-config-volume\") pod \"cd55d060-ec86-413c-8303-1d7bde266156\" (UID: \"cd55d060-ec86-413c-8303-1d7bde266156\") " Nov 03 21:07:49.045318 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:49.045292 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd55d060-ec86-413c-8303-1d7bde266156-plugins-config-volume" pod "cd55d060-ec86-413c-8303-1d7bde266156" (UID: "cd55d060-ec86-413c-8303-1d7bde266156"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:07:49.045502 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:49.045337 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd55d060-ec86-413c-8303-1d7bde266156-kube-api-access-t4r96\" (UniqueName: \"kubernetes.io/projected/cd55d060-ec86-413c-8303-1d7bde266156-kube-api-access-t4r96\") pod \"cd55d060-ec86-413c-8303-1d7bde266156\" (UID: \"cd55d060-ec86-413c-8303-1d7bde266156\") " Nov 03 21:07:49.045863 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:49.045836 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6b1772b-0599-4077-a0df-c21e82e80d97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0236317-jcldz\" (UID: \"b6b1772b-0599-4077-a0df-c21e82e80d97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0236317-jcldz" Nov 03 21:07:49.045996 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:49.045888 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqlzd\" (UniqueName: \"kubernetes.io/projected/b6b1772b-0599-4077-a0df-c21e82e80d97-kube-api-access-gqlzd\") pod \"maas-api-revoke-sub-f0236317-jcldz\" (UID: \"b6b1772b-0599-4077-a0df-c21e82e80d97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0236317-jcldz" Nov 03 21:07:49.046412 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:49.046387 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6b1772b-0599-4077-a0df-c21e82e80d97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0236317-jcldz\" (UID: \"b6b1772b-0599-4077-a0df-c21e82e80d97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0236317-jcldz" Nov 03 21:07:49.046631 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:49.046611 4305 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd55d060-ec86-413c-8303-1d7bde266156-plugins-config-volume\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:07:49.048135 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:49.048106 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd55d060-ec86-413c-8303-1d7bde266156-kube-api-access-t4r96" pod "cd55d060-ec86-413c-8303-1d7bde266156" (UID: "cd55d060-ec86-413c-8303-1d7bde266156"). InnerVolumeSpecName "kube-api-access-t4r96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:07:49.148203 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:49.148172 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6b1772b-0599-4077-a0df-c21e82e80d97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0236317-jcldz\" (UID: \"b6b1772b-0599-4077-a0df-c21e82e80d97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0236317-jcldz" Nov 03 21:07:49.148392 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:49.148281 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6b1772b-0599-4077-a0df-c21e82e80d97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0236317-jcldz\" (UID: \"b6b1772b-0599-4077-a0df-c21e82e80d97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0236317-jcldz" Nov 03 21:07:49.148392 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:49.148323 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqlzd\" (UniqueName: \"kubernetes.io/projected/b6b1772b-0599-4077-a0df-c21e82e80d97-kube-api-access-gqlzd\") pod \"maas-api-revoke-sub-f0236317-jcldz\" (UID: \"b6b1772b-0599-4077-a0df-c21e82e80d97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0236317-jcldz" Nov 03 21:07:49.148663 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:49.148644 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4r96\" (UniqueName: \"kubernetes.io/projected/cd55d060-ec86-413c-8303-1d7bde266156-kube-api-access-t4r96\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:07:49.149403 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:49.149374 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6b1772b-0599-4077-a0df-c21e82e80d97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0236317-jcldz\" (UID: \"b6b1772b-0599-4077-a0df-c21e82e80d97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0236317-jcldz" Nov 03 21:07:49.151002 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:49.150976 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6b1772b-0599-4077-a0df-c21e82e80d97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0236317-jcldz\" (UID: \"b6b1772b-0599-4077-a0df-c21e82e80d97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0236317-jcldz" Nov 03 21:07:49.210509 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:49.210475 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqlzd\" (UniqueName: \"kubernetes.io/projected/b6b1772b-0599-4077-a0df-c21e82e80d97-kube-api-access-gqlzd\") pod \"maas-api-revoke-sub-f0236317-jcldz\" (UID: \"b6b1772b-0599-4077-a0df-c21e82e80d97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0236317-jcldz" Nov 03 21:07:49.240524 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:49.240493 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0236317-jcldz" Nov 03 21:07:49.433110 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:49.433020 4305 generic.go:356] "Generic (PLEG): container finished" podID="cd55d060-ec86-413c-8303-1d7bde266156" containerID="8f55dcd9b33bd2860bc96e9dac6a1e017a8daeea77c363819a5e1ac956f85d24" exitCode=0 Nov 03 21:07:49.433110 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:49.433061 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e41159db-6967b48544-cwjl7" Nov 03 21:07:49.433366 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:49.433113 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e41159db-6967b48544-cwjl7" event={"ID":"cd55d060-ec86-413c-8303-1d7bde266156","Type":"ContainerDied","Data":"8f55dcd9b33bd2860bc96e9dac6a1e017a8daeea77c363819a5e1ac956f85d24"} Nov 03 21:07:49.433366 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:49.433175 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e41159db-6967b48544-cwjl7" event={"ID":"cd55d060-ec86-413c-8303-1d7bde266156","Type":"ContainerDied","Data":"6b5fba7333013b99d6b471c98ecab293f911e94dad679e760e8f0f8bf56afb7d"} Nov 03 21:07:49.433366 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:49.433207 4305 scope.go:122] "RemoveContainer" containerID="8f55dcd9b33bd2860bc96e9dac6a1e017a8daeea77c363819a5e1ac956f85d24" Nov 03 21:07:49.451667 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:49.451645 4305 scope.go:122] "RemoveContainer" containerID="8f55dcd9b33bd2860bc96e9dac6a1e017a8daeea77c363819a5e1ac956f85d24" Nov 03 21:07:49.452012 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:07:49.451987 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f55dcd9b33bd2860bc96e9dac6a1e017a8daeea77c363819a5e1ac956f85d24\": container with ID starting with 8f55dcd9b33bd2860bc96e9dac6a1e017a8daeea77c363819a5e1ac956f85d24 not found: ID does not exist" containerID="8f55dcd9b33bd2860bc96e9dac6a1e017a8daeea77c363819a5e1ac956f85d24" Nov 03 21:07:49.452081 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:49.452024 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f55dcd9b33bd2860bc96e9dac6a1e017a8daeea77c363819a5e1ac956f85d24"} err="failed to get container status \"8f55dcd9b33bd2860bc96e9dac6a1e017a8daeea77c363819a5e1ac956f85d24\": rpc error: code = NotFound desc = could not find container \"8f55dcd9b33bd2860bc96e9dac6a1e017a8daeea77c363819a5e1ac956f85d24\": container with ID starting with 8f55dcd9b33bd2860bc96e9dac6a1e017a8daeea77c363819a5e1ac956f85d24 not found: ID does not exist" Nov 03 21:07:49.602538 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:49.602510 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e41159db-6967b48544-cwjl7"] Nov 03 21:07:49.607599 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:49.607574 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e41159db-6967b48544-cwjl7"] Nov 03 21:07:49.649115 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:49.649086 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0236317-jcldz"] Nov 03 21:07:49.663409 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:07:49.663381 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6b1772b_0599_4077_a0df_c21e82e80d97.slice/crio-db79bee187c83c63579f5e4040dc2e8b3404d407737ccc1ff1209925bc7e548a WatchSource:0}: Error finding container db79bee187c83c63579f5e4040dc2e8b3404d407737ccc1ff1209925bc7e548a: Status 404 returned error can't find the container with id db79bee187c83c63579f5e4040dc2e8b3404d407737ccc1ff1209925bc7e548a Nov 03 21:07:50.458308 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:50.458265 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0236317-jcldz" event={"ID":"b6b1772b-0599-4077-a0df-c21e82e80d97","Type":"ContainerStarted","Data":"38897c2dd34e6fa54c70eb02bea42b5b623cfcdd4bb07c17be52d76a440fc8c7"} Nov 03 21:07:50.458518 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:50.458376 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0236317-jcldz" event={"ID":"b6b1772b-0599-4077-a0df-c21e82e80d97","Type":"ContainerStarted","Data":"db79bee187c83c63579f5e4040dc2e8b3404d407737ccc1ff1209925bc7e548a"} Nov 03 21:07:50.585913 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:50.585859 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0236317-jcldz" podStartSLOduration=2.585845926 podStartE2EDuration="2.585845926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:07:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:07:50.583223718 +0000 UTC m=+1705.808543521" watchObservedRunningTime="2026-11-03 21:07:50.585845926 +0000 UTC m=+1705.811165746" Nov 03 21:07:50.897855 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:50.897824 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd55d060-ec86-413c-8303-1d7bde266156" path="/var/lib/kubelet/pods/cd55d060-ec86-413c-8303-1d7bde266156/volumes" Nov 03 21:07:51.050413 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:51.050328 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2df98d9-8zlwf"] Nov 03 21:07:51.055022 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:51.054994 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2df98d9-8zlwf"] Nov 03 21:07:51.149128 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:51.149075 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-04f5216c-b64d9fdfb-p8dxz" podUID="e37f3c67-e93b-4933-b01e-22a99ef52532" containerName="payload-processing" containerID="cri-o://ce3ad58895272abc8d00cf473b893f260f5e4ba654aae6a07aec6c3bcf87424a" gracePeriod=25 Nov 03 21:07:51.227582 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:07:51.227552 4305 logging.go:55] [core] [Channel #3689 SubChannel #3692]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.160:50052", ServerName: "10.134.0.160:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.160:50052: connect: connection refused" Nov 03 21:07:51.249321 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:51.249263 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-04f5216c-fb66cddc-vbsb7" podUID="b3ff046c-42eb-40ce-b555-d069fc6a8567" containerName="payload-pre-processing" containerID="cri-o://6ca296fa79c740386c7f49a74094eabda5e56fcbb8b111084f4f679995908839" gracePeriod=25 Nov 03 21:07:51.446470 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:51.446444 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-04f5216c-b64d9fdfb-p8dxz" Nov 03 21:07:51.471207 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:51.471169 4305 generic.go:356] "Generic (PLEG): container finished" podID="b6b1772b-0599-4077-a0df-c21e82e80d97" containerID="38897c2dd34e6fa54c70eb02bea42b5b623cfcdd4bb07c17be52d76a440fc8c7" exitCode=0 Nov 03 21:07:51.471421 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:51.471333 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0236317-jcldz" event={"ID":"b6b1772b-0599-4077-a0df-c21e82e80d97","Type":"ContainerDied","Data":"38897c2dd34e6fa54c70eb02bea42b5b623cfcdd4bb07c17be52d76a440fc8c7"} Nov 03 21:07:51.473528 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:51.473499 4305 generic.go:356] "Generic (PLEG): container finished" podID="e37f3c67-e93b-4933-b01e-22a99ef52532" containerID="ce3ad58895272abc8d00cf473b893f260f5e4ba654aae6a07aec6c3bcf87424a" exitCode=0 Nov 03 21:07:51.473769 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:51.473750 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-04f5216c-b64d9fdfb-p8dxz" event={"ID":"e37f3c67-e93b-4933-b01e-22a99ef52532","Type":"ContainerDied","Data":"ce3ad58895272abc8d00cf473b893f260f5e4ba654aae6a07aec6c3bcf87424a"} Nov 03 21:07:51.473831 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:51.473799 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-04f5216c-b64d9fdfb-p8dxz" event={"ID":"e37f3c67-e93b-4933-b01e-22a99ef52532","Type":"ContainerDied","Data":"22d189cf0ed64d0b1d2fc5877274653df8f2ed892e3e87f93838cd65ddf546ca"} Nov 03 21:07:51.473831 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:51.473828 4305 scope.go:122] "RemoveContainer" containerID="ce3ad58895272abc8d00cf473b893f260f5e4ba654aae6a07aec6c3bcf87424a" Nov 03 21:07:51.474015 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:51.473998 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-04f5216c-b64d9fdfb-p8dxz" Nov 03 21:07:51.476307 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:51.476280 4305 generic.go:356] "Generic (PLEG): container finished" podID="b3ff046c-42eb-40ce-b555-d069fc6a8567" containerID="6ca296fa79c740386c7f49a74094eabda5e56fcbb8b111084f4f679995908839" exitCode=0 Nov 03 21:07:51.476489 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:51.476328 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-04f5216c-fb66cddc-vbsb7" event={"ID":"b3ff046c-42eb-40ce-b555-d069fc6a8567","Type":"ContainerDied","Data":"6ca296fa79c740386c7f49a74094eabda5e56fcbb8b111084f4f679995908839"} Nov 03 21:07:51.483236 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:51.483205 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e37f3c67-e93b-4933-b01e-22a99ef52532-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e37f3c67-e93b-4933-b01e-22a99ef52532-plugins-config-volume\") pod \"e37f3c67-e93b-4933-b01e-22a99ef52532\" (UID: \"e37f3c67-e93b-4933-b01e-22a99ef52532\") " Nov 03 21:07:51.483511 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:51.483491 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e37f3c67-e93b-4933-b01e-22a99ef52532-kube-api-access-mnzbm\" (UniqueName: \"kubernetes.io/projected/e37f3c67-e93b-4933-b01e-22a99ef52532-kube-api-access-mnzbm\") pod \"e37f3c67-e93b-4933-b01e-22a99ef52532\" (UID: \"e37f3c67-e93b-4933-b01e-22a99ef52532\") " Nov 03 21:07:51.485347 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:51.485248 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e37f3c67-e93b-4933-b01e-22a99ef52532-plugins-config-volume" pod "e37f3c67-e93b-4933-b01e-22a99ef52532" (UID: "e37f3c67-e93b-4933-b01e-22a99ef52532"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:07:51.486731 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:51.486704 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e37f3c67-e93b-4933-b01e-22a99ef52532-kube-api-access-mnzbm" pod "e37f3c67-e93b-4933-b01e-22a99ef52532" (UID: "e37f3c67-e93b-4933-b01e-22a99ef52532"). InnerVolumeSpecName "kube-api-access-mnzbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:07:51.488136 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:51.488114 4305 scope.go:122] "RemoveContainer" containerID="ce3ad58895272abc8d00cf473b893f260f5e4ba654aae6a07aec6c3bcf87424a" Nov 03 21:07:51.488556 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:07:51.488527 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce3ad58895272abc8d00cf473b893f260f5e4ba654aae6a07aec6c3bcf87424a\": container with ID starting with ce3ad58895272abc8d00cf473b893f260f5e4ba654aae6a07aec6c3bcf87424a not found: ID does not exist" containerID="ce3ad58895272abc8d00cf473b893f260f5e4ba654aae6a07aec6c3bcf87424a" Nov 03 21:07:51.488658 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:51.488565 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce3ad58895272abc8d00cf473b893f260f5e4ba654aae6a07aec6c3bcf87424a"} err="failed to get container status \"ce3ad58895272abc8d00cf473b893f260f5e4ba654aae6a07aec6c3bcf87424a\": rpc error: code = NotFound desc = could not find container \"ce3ad58895272abc8d00cf473b893f260f5e4ba654aae6a07aec6c3bcf87424a\": container with ID starting with ce3ad58895272abc8d00cf473b893f260f5e4ba654aae6a07aec6c3bcf87424a not found: ID does not exist" Nov 03 21:07:51.543358 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:51.543332 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-04f5216c-fb66cddc-vbsb7" Nov 03 21:07:51.585776 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:51.585374 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3ff046c-42eb-40ce-b555-d069fc6a8567-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3ff046c-42eb-40ce-b555-d069fc6a8567-plugins-config-volume\") pod \"b3ff046c-42eb-40ce-b555-d069fc6a8567\" (UID: \"b3ff046c-42eb-40ce-b555-d069fc6a8567\") " Nov 03 21:07:51.585776 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:51.585513 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3ff046c-42eb-40ce-b555-d069fc6a8567-kube-api-access-7jdr2\" (UniqueName: \"kubernetes.io/projected/b3ff046c-42eb-40ce-b555-d069fc6a8567-kube-api-access-7jdr2\") pod \"b3ff046c-42eb-40ce-b555-d069fc6a8567\" (UID: \"b3ff046c-42eb-40ce-b555-d069fc6a8567\") " Nov 03 21:07:51.586205 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:51.586174 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3ff046c-42eb-40ce-b555-d069fc6a8567-plugins-config-volume" pod "b3ff046c-42eb-40ce-b555-d069fc6a8567" (UID: "b3ff046c-42eb-40ce-b555-d069fc6a8567"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:07:51.586340 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:51.586323 4305 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e37f3c67-e93b-4933-b01e-22a99ef52532-plugins-config-volume\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:07:51.586403 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:51.586344 4305 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3ff046c-42eb-40ce-b555-d069fc6a8567-plugins-config-volume\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:07:51.586403 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:51.586359 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnzbm\" (UniqueName: \"kubernetes.io/projected/e37f3c67-e93b-4933-b01e-22a99ef52532-kube-api-access-mnzbm\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:07:51.588577 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:51.588550 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3ff046c-42eb-40ce-b555-d069fc6a8567-kube-api-access-7jdr2" pod "b3ff046c-42eb-40ce-b555-d069fc6a8567" (UID: "b3ff046c-42eb-40ce-b555-d069fc6a8567"). InnerVolumeSpecName "kube-api-access-7jdr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:07:51.688837 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:51.688784 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jdr2\" (UniqueName: \"kubernetes.io/projected/b3ff046c-42eb-40ce-b555-d069fc6a8567-kube-api-access-7jdr2\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:07:51.961049 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:51.961016 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-04f5216c-b64d9fdfb-p8dxz"] Nov 03 21:07:51.969847 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:51.969818 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-04f5216c-b64d9fdfb-p8dxz"] Nov 03 21:07:52.228315 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:52.228226 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ait-04f5216c-b64d9fdfb-p8dxz" podUID="e37f3c67-e93b-4933-b01e-22a99ef52532" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.160:50052\" within 1s: context deadline exceeded" Nov 03 21:07:52.228502 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:07:52.228346 4305 logging.go:55] [core] [Channel #3689 SubChannel #3692]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.160:50052", ServerName: "10.134.0.160:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.160:50052: operation was canceled" Nov 03 21:07:52.485693 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:52.485619 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-04f5216c-fb66cddc-vbsb7" Nov 03 21:07:52.485693 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:52.485659 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-04f5216c-fb66cddc-vbsb7" event={"ID":"b3ff046c-42eb-40ce-b555-d069fc6a8567","Type":"ContainerDied","Data":"3cd3da36e6f6633b16cef89adac0ffbea07fca00df18b2426614b50b61a10245"} Nov 03 21:07:52.485916 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:52.485758 4305 scope.go:122] "RemoveContainer" containerID="6ca296fa79c740386c7f49a74094eabda5e56fcbb8b111084f4f679995908839" Nov 03 21:07:52.669824 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:52.669785 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-04f5216c-fb66cddc-vbsb7"] Nov 03 21:07:52.675167 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:52.675142 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-04f5216c-fb66cddc-vbsb7"] Nov 03 21:07:52.676348 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:52.676330 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0236317-jcldz" Nov 03 21:07:52.704821 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:52.704784 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6b1772b-0599-4077-a0df-c21e82e80d97-kube-api-access-gqlzd\" (UniqueName: \"kubernetes.io/projected/b6b1772b-0599-4077-a0df-c21e82e80d97-kube-api-access-gqlzd\") pod \"b6b1772b-0599-4077-a0df-c21e82e80d97\" (UID: \"b6b1772b-0599-4077-a0df-c21e82e80d97\") " Nov 03 21:07:52.705023 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:52.705004 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6b1772b-0599-4077-a0df-c21e82e80d97-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6b1772b-0599-4077-a0df-c21e82e80d97-maas-api-openshift-ca\") pod \"b6b1772b-0599-4077-a0df-c21e82e80d97\" (UID: \"b6b1772b-0599-4077-a0df-c21e82e80d97\") " Nov 03 21:07:52.705154 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:52.705130 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6b1772b-0599-4077-a0df-c21e82e80d97-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6b1772b-0599-4077-a0df-c21e82e80d97-maas-api-xks-ca\") pod \"b6b1772b-0599-4077-a0df-c21e82e80d97\" (UID: \"b6b1772b-0599-4077-a0df-c21e82e80d97\") " Nov 03 21:07:52.705420 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:52.705394 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6b1772b-0599-4077-a0df-c21e82e80d97-maas-api-openshift-ca" pod "b6b1772b-0599-4077-a0df-c21e82e80d97" (UID: "b6b1772b-0599-4077-a0df-c21e82e80d97"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:07:52.705831 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:52.705813 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6b1772b-0599-4077-a0df-c21e82e80d97-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:07:52.707301 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:52.707276 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6b1772b-0599-4077-a0df-c21e82e80d97-kube-api-access-gqlzd" pod "b6b1772b-0599-4077-a0df-c21e82e80d97" (UID: "b6b1772b-0599-4077-a0df-c21e82e80d97"). InnerVolumeSpecName "kube-api-access-gqlzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:07:52.707414 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:52.707333 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6b1772b-0599-4077-a0df-c21e82e80d97-maas-api-xks-ca" pod "b6b1772b-0599-4077-a0df-c21e82e80d97" (UID: "b6b1772b-0599-4077-a0df-c21e82e80d97"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:07:52.807485 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:52.807359 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqlzd\" (UniqueName: \"kubernetes.io/projected/b6b1772b-0599-4077-a0df-c21e82e80d97-kube-api-access-gqlzd\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:07:52.807485 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:52.807401 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6b1772b-0599-4077-a0df-c21e82e80d97-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:07:52.898020 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:52.897985 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cff10ef-287d-40dd-a147-2b928fc7d053" path="/var/lib/kubelet/pods/2cff10ef-287d-40dd-a147-2b928fc7d053/volumes" Nov 03 21:07:52.898495 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:52.898478 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3ff046c-42eb-40ce-b555-d069fc6a8567" path="/var/lib/kubelet/pods/b3ff046c-42eb-40ce-b555-d069fc6a8567/volumes" Nov 03 21:07:52.898961 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:52.898947 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e37f3c67-e93b-4933-b01e-22a99ef52532" path="/var/lib/kubelet/pods/e37f3c67-e93b-4933-b01e-22a99ef52532/volumes" Nov 03 21:07:53.498471 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:53.498415 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0236317-jcldz" event={"ID":"b6b1772b-0599-4077-a0df-c21e82e80d97","Type":"ContainerDied","Data":"db79bee187c83c63579f5e4040dc2e8b3404d407737ccc1ff1209925bc7e548a"} Nov 03 21:07:53.499016 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:53.498508 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db79bee187c83c63579f5e4040dc2e8b3404d407737ccc1ff1209925bc7e548a" Nov 03 21:07:53.499016 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:53.498506 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0236317-jcldz" Nov 03 21:07:54.049925 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:54.049894 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d830196-vhlrr"] Nov 03 21:07:54.100094 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:54.100060 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d830196-vhlrr"] Nov 03 21:07:54.104310 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:54.104280 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j"] Nov 03 21:07:54.104540 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:54.104517 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j" podUID="2366aada-a95b-404a-b5a7-454a1670c687" containerName="istio-proxy" containerID="cri-o://0486deee2a7aa8f3c903d1dc0ebf3fd12299e62bcc6f18296e98eb9b34718bf2" gracePeriod=30 Nov 03 21:07:54.897866 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:54.897834 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="948e2234-c69a-47c6-b4f1-6d61d8ac19dd" path="/var/lib/kubelet/pods/948e2234-c69a-47c6-b4f1-6d61d8ac19dd/volumes" Nov 03 21:07:58.050615 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:58.050583 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4ad2ebc-hxffw"] Nov 03 21:07:58.098929 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:58.098888 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4ad2ebc-hxffw"] Nov 03 21:07:58.897453 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:58.897411 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b7fadf9-8365-466e-925f-c4fb8ff49955" path="/var/lib/kubelet/pods/6b7fadf9-8365-466e-925f-c4fb8ff49955/volumes" Nov 03 21:07:59.560291 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:59.560258 4305 generic.go:356] "Generic (PLEG): container finished" podID="2366aada-a95b-404a-b5a7-454a1670c687" containerID="0486deee2a7aa8f3c903d1dc0ebf3fd12299e62bcc6f18296e98eb9b34718bf2" exitCode=0 Nov 03 21:07:59.560739 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:59.560335 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j" event={"ID":"2366aada-a95b-404a-b5a7-454a1670c687","Type":"ContainerDied","Data":"0486deee2a7aa8f3c903d1dc0ebf3fd12299e62bcc6f18296e98eb9b34718bf2"} Nov 03 21:07:59.599174 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:59.599148 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j" Nov 03 21:07:59.633387 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:59.633355 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:07:59.734148 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:59.734064 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2366aada-a95b-404a-b5a7-454a1670c687-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2366aada-a95b-404a-b5a7-454a1670c687-istiod-ca-cert\") pod \"2366aada-a95b-404a-b5a7-454a1670c687\" (UID: \"2366aada-a95b-404a-b5a7-454a1670c687\") " Nov 03 21:07:59.734323 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:59.734152 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2366aada-a95b-404a-b5a7-454a1670c687-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2366aada-a95b-404a-b5a7-454a1670c687-istio-podinfo\") pod \"2366aada-a95b-404a-b5a7-454a1670c687\" (UID: \"2366aada-a95b-404a-b5a7-454a1670c687\") " Nov 03 21:07:59.734323 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:59.734232 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2366aada-a95b-404a-b5a7-454a1670c687-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2366aada-a95b-404a-b5a7-454a1670c687-istio-data\") pod \"2366aada-a95b-404a-b5a7-454a1670c687\" (UID: \"2366aada-a95b-404a-b5a7-454a1670c687\") " Nov 03 21:07:59.734323 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:59.734267 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2366aada-a95b-404a-b5a7-454a1670c687-istio-token\" (UniqueName: \"kubernetes.io/projected/2366aada-a95b-404a-b5a7-454a1670c687-istio-token\") pod \"2366aada-a95b-404a-b5a7-454a1670c687\" (UID: \"2366aada-a95b-404a-b5a7-454a1670c687\") " Nov 03 21:07:59.734638 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:59.734609 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2366aada-a95b-404a-b5a7-454a1670c687-istio-data" pod "2366aada-a95b-404a-b5a7-454a1670c687" (UID: "2366aada-a95b-404a-b5a7-454a1670c687"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:07:59.734717 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:59.734640 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2366aada-a95b-404a-b5a7-454a1670c687-istiod-ca-cert" pod "2366aada-a95b-404a-b5a7-454a1670c687" (UID: "2366aada-a95b-404a-b5a7-454a1670c687"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:07:59.734780 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:59.734761 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2366aada-a95b-404a-b5a7-454a1670c687-kube-api-access-4z4kr\" (UniqueName: \"kubernetes.io/projected/2366aada-a95b-404a-b5a7-454a1670c687-kube-api-access-4z4kr\") pod \"2366aada-a95b-404a-b5a7-454a1670c687\" (UID: \"2366aada-a95b-404a-b5a7-454a1670c687\") " Nov 03 21:07:59.734929 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:59.734911 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2366aada-a95b-404a-b5a7-454a1670c687-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2366aada-a95b-404a-b5a7-454a1670c687-istio-envoy\") pod \"2366aada-a95b-404a-b5a7-454a1670c687\" (UID: \"2366aada-a95b-404a-b5a7-454a1670c687\") " Nov 03 21:07:59.735031 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:59.735009 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2366aada-a95b-404a-b5a7-454a1670c687-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2366aada-a95b-404a-b5a7-454a1670c687-workload-certs\") pod \"2366aada-a95b-404a-b5a7-454a1670c687\" (UID: \"2366aada-a95b-404a-b5a7-454a1670c687\") " Nov 03 21:07:59.735482 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:59.735413 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2366aada-a95b-404a-b5a7-454a1670c687-workload-certs" pod "2366aada-a95b-404a-b5a7-454a1670c687" (UID: "2366aada-a95b-404a-b5a7-454a1670c687"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:07:59.735587 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:59.735525 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2366aada-a95b-404a-b5a7-454a1670c687-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2366aada-a95b-404a-b5a7-454a1670c687-workload-socket\") pod \"2366aada-a95b-404a-b5a7-454a1670c687\" (UID: \"2366aada-a95b-404a-b5a7-454a1670c687\") " Nov 03 21:07:59.735652 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:59.735631 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2366aada-a95b-404a-b5a7-454a1670c687-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2366aada-a95b-404a-b5a7-454a1670c687-credential-socket\") pod \"2366aada-a95b-404a-b5a7-454a1670c687\" (UID: \"2366aada-a95b-404a-b5a7-454a1670c687\") " Nov 03 21:07:59.737322 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:59.736680 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2366aada-a95b-404a-b5a7-454a1670c687-credential-socket" pod "2366aada-a95b-404a-b5a7-454a1670c687" (UID: "2366aada-a95b-404a-b5a7-454a1670c687"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:07:59.737322 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:59.736978 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2366aada-a95b-404a-b5a7-454a1670c687-istio-data\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:07:59.737322 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:59.736999 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2366aada-a95b-404a-b5a7-454a1670c687-workload-certs\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:07:59.737322 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:59.737036 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2366aada-a95b-404a-b5a7-454a1670c687-istiod-ca-cert\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:07:59.737322 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:59.737220 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2366aada-a95b-404a-b5a7-454a1670c687-istio-token" pod "2366aada-a95b-404a-b5a7-454a1670c687" (UID: "2366aada-a95b-404a-b5a7-454a1670c687"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:07:59.737322 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:59.737293 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2366aada-a95b-404a-b5a7-454a1670c687-workload-socket" pod "2366aada-a95b-404a-b5a7-454a1670c687" (UID: "2366aada-a95b-404a-b5a7-454a1670c687"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:07:59.738871 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:59.738828 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2366aada-a95b-404a-b5a7-454a1670c687-istio-envoy" pod "2366aada-a95b-404a-b5a7-454a1670c687" (UID: "2366aada-a95b-404a-b5a7-454a1670c687"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:07:59.738988 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:59.738965 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2366aada-a95b-404a-b5a7-454a1670c687-kube-api-access-4z4kr" pod "2366aada-a95b-404a-b5a7-454a1670c687" (UID: "2366aada-a95b-404a-b5a7-454a1670c687"). InnerVolumeSpecName "kube-api-access-4z4kr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:07:59.752640 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:59.752566 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2366aada-a95b-404a-b5a7-454a1670c687-istio-podinfo" pod "2366aada-a95b-404a-b5a7-454a1670c687" (UID: "2366aada-a95b-404a-b5a7-454a1670c687"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:07:59.840312 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:59.838914 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2366aada-a95b-404a-b5a7-454a1670c687-istio-envoy\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:07:59.840312 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:59.838952 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2366aada-a95b-404a-b5a7-454a1670c687-workload-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:07:59.840312 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:59.838968 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2366aada-a95b-404a-b5a7-454a1670c687-credential-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:07:59.840312 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:59.838981 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2366aada-a95b-404a-b5a7-454a1670c687-istio-podinfo\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:07:59.840312 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:59.838995 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2366aada-a95b-404a-b5a7-454a1670c687-istio-token\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:07:59.840312 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:07:59.839008 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z4kr\" (UniqueName: \"kubernetes.io/projected/2366aada-a95b-404a-b5a7-454a1670c687-kube-api-access-4z4kr\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:08:00.570630 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:00.570602 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j" Nov 03 21:08:00.570630 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:00.570615 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j" event={"ID":"2366aada-a95b-404a-b5a7-454a1670c687","Type":"ContainerDied","Data":"bc86ac6cdaa9cf246dcd9a323992a7701c4a7b6fe8cf804f39f4ffb98d36448e"} Nov 03 21:08:00.571165 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:00.570687 4305 scope.go:122] "RemoveContainer" containerID="0486deee2a7aa8f3c903d1dc0ebf3fd12299e62bcc6f18296e98eb9b34718bf2" Nov 03 21:08:00.761382 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:00.755497 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j"] Nov 03 21:08:00.767924 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:00.767885 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-nvl7j"] Nov 03 21:08:00.911766 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:00.910928 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2366aada-a95b-404a-b5a7-454a1670c687" path="/var/lib/kubelet/pods/2366aada-a95b-404a-b5a7-454a1670c687/volumes" Nov 03 21:08:03.168574 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:03.168540 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e3917193-mpsw2"] Nov 03 21:08:03.218097 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:03.218056 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e3917193-mpsw2"] Nov 03 21:08:04.898134 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:04.898102 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a170fb6b-d3eb-474a-a4cc-1fced5daf1c5" path="/var/lib/kubelet/pods/a170fb6b-d3eb-474a-a4cc-1fced5daf1c5/volumes" Nov 03 21:08:10.023561 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:10.023524 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz"] Nov 03 21:08:10.024168 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:10.023810 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz" podUID="95dd3f17-e145-49ec-8fe2-6148c3f07c80" containerName="istio-proxy" containerID="cri-o://255deffa733f6070921d532a2bbcd6bccb113337efaa216e7d3027b21b4e7357" gracePeriod=30 Nov 03 21:08:10.372960 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:10.372861 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz"] Nov 03 21:08:10.373178 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:10.373149 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz" podUID="f5b60f69-9404-49bd-a072-d04f9b4ccb19" containerName="istio-proxy" containerID="cri-o://bb9e0ba1ad5778ed22016235710eb53c0092ea96b8929007d2495fe00e69ae13" gracePeriod=30 Nov 03 21:08:15.159650 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.159616 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e41159db-6l7cz"] Nov 03 21:08:15.221987 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.210303 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e41159db-6l7cz"] Nov 03 21:08:15.325923 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.325893 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz" Nov 03 21:08:15.464957 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.464923 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95dd3f17-e145-49ec-8fe2-6148c3f07c80-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95dd3f17-e145-49ec-8fe2-6148c3f07c80-workload-certs\") pod \"95dd3f17-e145-49ec-8fe2-6148c3f07c80\" (UID: \"95dd3f17-e145-49ec-8fe2-6148c3f07c80\") " Nov 03 21:08:15.465156 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.465079 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95dd3f17-e145-49ec-8fe2-6148c3f07c80-kube-api-access-8wvw5\" (UniqueName: \"kubernetes.io/projected/95dd3f17-e145-49ec-8fe2-6148c3f07c80-kube-api-access-8wvw5\") pod \"95dd3f17-e145-49ec-8fe2-6148c3f07c80\" (UID: \"95dd3f17-e145-49ec-8fe2-6148c3f07c80\") " Nov 03 21:08:15.465224 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.465199 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95dd3f17-e145-49ec-8fe2-6148c3f07c80-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95dd3f17-e145-49ec-8fe2-6148c3f07c80-credential-socket\") pod \"95dd3f17-e145-49ec-8fe2-6148c3f07c80\" (UID: \"95dd3f17-e145-49ec-8fe2-6148c3f07c80\") " Nov 03 21:08:15.465286 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.465258 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95dd3f17-e145-49ec-8fe2-6148c3f07c80-workload-certs" pod "95dd3f17-e145-49ec-8fe2-6148c3f07c80" (UID: "95dd3f17-e145-49ec-8fe2-6148c3f07c80"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:08:15.465469 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.465447 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95dd3f17-e145-49ec-8fe2-6148c3f07c80-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95dd3f17-e145-49ec-8fe2-6148c3f07c80-workload-socket\") pod \"95dd3f17-e145-49ec-8fe2-6148c3f07c80\" (UID: \"95dd3f17-e145-49ec-8fe2-6148c3f07c80\") " Nov 03 21:08:15.465673 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.465530 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95dd3f17-e145-49ec-8fe2-6148c3f07c80-credential-socket" pod "95dd3f17-e145-49ec-8fe2-6148c3f07c80" (UID: "95dd3f17-e145-49ec-8fe2-6148c3f07c80"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:08:15.465749 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.465653 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95dd3f17-e145-49ec-8fe2-6148c3f07c80-workload-socket" pod "95dd3f17-e145-49ec-8fe2-6148c3f07c80" (UID: "95dd3f17-e145-49ec-8fe2-6148c3f07c80"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:08:15.465899 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.465875 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95dd3f17-e145-49ec-8fe2-6148c3f07c80-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95dd3f17-e145-49ec-8fe2-6148c3f07c80-istio-data\") pod \"95dd3f17-e145-49ec-8fe2-6148c3f07c80\" (UID: \"95dd3f17-e145-49ec-8fe2-6148c3f07c80\") " Nov 03 21:08:15.466025 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.465931 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/95dd3f17-e145-49ec-8fe2-6148c3f07c80-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95dd3f17-e145-49ec-8fe2-6148c3f07c80-istio-podinfo\") pod \"95dd3f17-e145-49ec-8fe2-6148c3f07c80\" (UID: \"95dd3f17-e145-49ec-8fe2-6148c3f07c80\") " Nov 03 21:08:15.466180 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.466164 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95dd3f17-e145-49ec-8fe2-6148c3f07c80-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95dd3f17-e145-49ec-8fe2-6148c3f07c80-istiod-ca-cert\") pod \"95dd3f17-e145-49ec-8fe2-6148c3f07c80\" (UID: \"95dd3f17-e145-49ec-8fe2-6148c3f07c80\") " Nov 03 21:08:15.466270 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.466251 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95dd3f17-e145-49ec-8fe2-6148c3f07c80-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95dd3f17-e145-49ec-8fe2-6148c3f07c80-istio-envoy\") pod \"95dd3f17-e145-49ec-8fe2-6148c3f07c80\" (UID: \"95dd3f17-e145-49ec-8fe2-6148c3f07c80\") " Nov 03 21:08:15.466365 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.466308 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95dd3f17-e145-49ec-8fe2-6148c3f07c80-istio-token\" (UniqueName: \"kubernetes.io/projected/95dd3f17-e145-49ec-8fe2-6148c3f07c80-istio-token\") pod \"95dd3f17-e145-49ec-8fe2-6148c3f07c80\" (UID: \"95dd3f17-e145-49ec-8fe2-6148c3f07c80\") " Nov 03 21:08:15.466631 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.466415 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95dd3f17-e145-49ec-8fe2-6148c3f07c80-istio-data" pod "95dd3f17-e145-49ec-8fe2-6148c3f07c80" (UID: "95dd3f17-e145-49ec-8fe2-6148c3f07c80"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:08:15.467099 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.466859 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95dd3f17-e145-49ec-8fe2-6148c3f07c80-istiod-ca-cert" pod "95dd3f17-e145-49ec-8fe2-6148c3f07c80" (UID: "95dd3f17-e145-49ec-8fe2-6148c3f07c80"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:08:15.467293 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.467267 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95dd3f17-e145-49ec-8fe2-6148c3f07c80-credential-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:08:15.467360 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.467301 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95dd3f17-e145-49ec-8fe2-6148c3f07c80-workload-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:08:15.467360 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.467316 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95dd3f17-e145-49ec-8fe2-6148c3f07c80-istio-data\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:08:15.467360 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.467330 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95dd3f17-e145-49ec-8fe2-6148c3f07c80-istiod-ca-cert\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:08:15.467360 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.467344 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95dd3f17-e145-49ec-8fe2-6148c3f07c80-workload-certs\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:08:15.468866 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.468835 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95dd3f17-e145-49ec-8fe2-6148c3f07c80-kube-api-access-8wvw5" pod "95dd3f17-e145-49ec-8fe2-6148c3f07c80" (UID: "95dd3f17-e145-49ec-8fe2-6148c3f07c80"). InnerVolumeSpecName "kube-api-access-8wvw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:08:15.469183 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.469161 4305 patch_prober.go:29] interesting pod/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 21:08:15.469265 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.469224 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz" podUID="f5b60f69-9404-49bd-a072-d04f9b4ccb19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 21:08:15.471589 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.471564 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/95dd3f17-e145-49ec-8fe2-6148c3f07c80-istio-podinfo" pod "95dd3f17-e145-49ec-8fe2-6148c3f07c80" (UID: "95dd3f17-e145-49ec-8fe2-6148c3f07c80"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:08:15.471589 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.471577 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95dd3f17-e145-49ec-8fe2-6148c3f07c80-istio-envoy" pod "95dd3f17-e145-49ec-8fe2-6148c3f07c80" (UID: "95dd3f17-e145-49ec-8fe2-6148c3f07c80"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:08:15.471723 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.471593 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95dd3f17-e145-49ec-8fe2-6148c3f07c80-istio-token" pod "95dd3f17-e145-49ec-8fe2-6148c3f07c80" (UID: "95dd3f17-e145-49ec-8fe2-6148c3f07c80"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:08:15.572099 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.570188 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95dd3f17-e145-49ec-8fe2-6148c3f07c80-istio-podinfo\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:08:15.572099 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.570227 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95dd3f17-e145-49ec-8fe2-6148c3f07c80-istio-envoy\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:08:15.572099 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.570241 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95dd3f17-e145-49ec-8fe2-6148c3f07c80-istio-token\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:08:15.572099 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.570255 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wvw5\" (UniqueName: \"kubernetes.io/projected/95dd3f17-e145-49ec-8fe2-6148c3f07c80-kube-api-access-8wvw5\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:08:15.644374 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.644346 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz" Nov 03 21:08:15.730413 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.730373 4305 generic.go:356] "Generic (PLEG): container finished" podID="95dd3f17-e145-49ec-8fe2-6148c3f07c80" containerID="255deffa733f6070921d532a2bbcd6bccb113337efaa216e7d3027b21b4e7357" exitCode=0 Nov 03 21:08:15.730648 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.730472 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz" Nov 03 21:08:15.730648 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.730473 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz" event={"ID":"95dd3f17-e145-49ec-8fe2-6148c3f07c80","Type":"ContainerDied","Data":"255deffa733f6070921d532a2bbcd6bccb113337efaa216e7d3027b21b4e7357"} Nov 03 21:08:15.730648 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.730611 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz" event={"ID":"95dd3f17-e145-49ec-8fe2-6148c3f07c80","Type":"ContainerDied","Data":"7e03790f025b0417c6e4f29adef4d514bb01facba5538d293177946e16b193d4"} Nov 03 21:08:15.730648 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.730644 4305 scope.go:122] "RemoveContainer" containerID="255deffa733f6070921d532a2bbcd6bccb113337efaa216e7d3027b21b4e7357" Nov 03 21:08:15.734025 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.733998 4305 generic.go:356] "Generic (PLEG): container finished" podID="f5b60f69-9404-49bd-a072-d04f9b4ccb19" containerID="bb9e0ba1ad5778ed22016235710eb53c0092ea96b8929007d2495fe00e69ae13" exitCode=0 Nov 03 21:08:15.734137 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.734053 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz" Nov 03 21:08:15.734137 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.734088 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz" event={"ID":"f5b60f69-9404-49bd-a072-d04f9b4ccb19","Type":"ContainerDied","Data":"bb9e0ba1ad5778ed22016235710eb53c0092ea96b8929007d2495fe00e69ae13"} Nov 03 21:08:15.734223 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.734147 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz" event={"ID":"f5b60f69-9404-49bd-a072-d04f9b4ccb19","Type":"ContainerDied","Data":"e9f777fbf30b9ec20089e6f5255d90b9dd010f65b5bca1bd0a57c889ccef9d54"} Nov 03 21:08:15.743384 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.743361 4305 scope.go:122] "RemoveContainer" containerID="18c6ebd03bc3927d7424bc82cfe496702ce416880bdeeaa6523167f626cbe951" Nov 03 21:08:15.754968 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.754945 4305 scope.go:122] "RemoveContainer" containerID="255deffa733f6070921d532a2bbcd6bccb113337efaa216e7d3027b21b4e7357" Nov 03 21:08:15.755318 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:08:15.755293 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"255deffa733f6070921d532a2bbcd6bccb113337efaa216e7d3027b21b4e7357\": container with ID starting with 255deffa733f6070921d532a2bbcd6bccb113337efaa216e7d3027b21b4e7357 not found: ID does not exist" containerID="255deffa733f6070921d532a2bbcd6bccb113337efaa216e7d3027b21b4e7357" Nov 03 21:08:15.755407 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.755329 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"255deffa733f6070921d532a2bbcd6bccb113337efaa216e7d3027b21b4e7357"} err="failed to get container status \"255deffa733f6070921d532a2bbcd6bccb113337efaa216e7d3027b21b4e7357\": rpc error: code = NotFound desc = could not find container \"255deffa733f6070921d532a2bbcd6bccb113337efaa216e7d3027b21b4e7357\": container with ID starting with 255deffa733f6070921d532a2bbcd6bccb113337efaa216e7d3027b21b4e7357 not found: ID does not exist" Nov 03 21:08:15.755407 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.755348 4305 scope.go:122] "RemoveContainer" containerID="18c6ebd03bc3927d7424bc82cfe496702ce416880bdeeaa6523167f626cbe951" Nov 03 21:08:15.755715 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:08:15.755694 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18c6ebd03bc3927d7424bc82cfe496702ce416880bdeeaa6523167f626cbe951\": container with ID starting with 18c6ebd03bc3927d7424bc82cfe496702ce416880bdeeaa6523167f626cbe951 not found: ID does not exist" containerID="18c6ebd03bc3927d7424bc82cfe496702ce416880bdeeaa6523167f626cbe951" Nov 03 21:08:15.755753 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.755723 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18c6ebd03bc3927d7424bc82cfe496702ce416880bdeeaa6523167f626cbe951"} err="failed to get container status \"18c6ebd03bc3927d7424bc82cfe496702ce416880bdeeaa6523167f626cbe951\": rpc error: code = NotFound desc = could not find container \"18c6ebd03bc3927d7424bc82cfe496702ce416880bdeeaa6523167f626cbe951\": container with ID starting with 18c6ebd03bc3927d7424bc82cfe496702ce416880bdeeaa6523167f626cbe951 not found: ID does not exist" Nov 03 21:08:15.755753 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.755741 4305 scope.go:122] "RemoveContainer" containerID="bb9e0ba1ad5778ed22016235710eb53c0092ea96b8929007d2495fe00e69ae13" Nov 03 21:08:15.767308 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.767288 4305 scope.go:122] "RemoveContainer" containerID="bb9e0ba1ad5778ed22016235710eb53c0092ea96b8929007d2495fe00e69ae13" Nov 03 21:08:15.767662 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:08:15.767643 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb9e0ba1ad5778ed22016235710eb53c0092ea96b8929007d2495fe00e69ae13\": container with ID starting with bb9e0ba1ad5778ed22016235710eb53c0092ea96b8929007d2495fe00e69ae13 not found: ID does not exist" containerID="bb9e0ba1ad5778ed22016235710eb53c0092ea96b8929007d2495fe00e69ae13" Nov 03 21:08:15.767736 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.767670 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb9e0ba1ad5778ed22016235710eb53c0092ea96b8929007d2495fe00e69ae13"} err="failed to get container status \"bb9e0ba1ad5778ed22016235710eb53c0092ea96b8929007d2495fe00e69ae13\": rpc error: code = NotFound desc = could not find container \"bb9e0ba1ad5778ed22016235710eb53c0092ea96b8929007d2495fe00e69ae13\": container with ID starting with bb9e0ba1ad5778ed22016235710eb53c0092ea96b8929007d2495fe00e69ae13 not found: ID does not exist" Nov 03 21:08:15.773322 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.773298 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5b60f69-9404-49bd-a072-d04f9b4ccb19-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5b60f69-9404-49bd-a072-d04f9b4ccb19-istio-envoy\") pod \"f5b60f69-9404-49bd-a072-d04f9b4ccb19\" (UID: \"f5b60f69-9404-49bd-a072-d04f9b4ccb19\") " Nov 03 21:08:15.773451 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.773361 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5b60f69-9404-49bd-a072-d04f9b4ccb19-istio-token\" (UniqueName: \"kubernetes.io/projected/f5b60f69-9404-49bd-a072-d04f9b4ccb19-istio-token\") pod \"f5b60f69-9404-49bd-a072-d04f9b4ccb19\" (UID: \"f5b60f69-9404-49bd-a072-d04f9b4ccb19\") " Nov 03 21:08:15.773451 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.773389 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f5b60f69-9404-49bd-a072-d04f9b4ccb19-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5b60f69-9404-49bd-a072-d04f9b4ccb19-istio-podinfo\") pod \"f5b60f69-9404-49bd-a072-d04f9b4ccb19\" (UID: \"f5b60f69-9404-49bd-a072-d04f9b4ccb19\") " Nov 03 21:08:15.773451 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.773420 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5b60f69-9404-49bd-a072-d04f9b4ccb19-kube-api-access-cjg42\" (UniqueName: \"kubernetes.io/projected/f5b60f69-9404-49bd-a072-d04f9b4ccb19-kube-api-access-cjg42\") pod \"f5b60f69-9404-49bd-a072-d04f9b4ccb19\" (UID: \"f5b60f69-9404-49bd-a072-d04f9b4ccb19\") " Nov 03 21:08:15.773684 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.773659 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5b60f69-9404-49bd-a072-d04f9b4ccb19-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5b60f69-9404-49bd-a072-d04f9b4ccb19-istio-data\") pod \"f5b60f69-9404-49bd-a072-d04f9b4ccb19\" (UID: \"f5b60f69-9404-49bd-a072-d04f9b4ccb19\") " Nov 03 21:08:15.773862 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.773817 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5b60f69-9404-49bd-a072-d04f9b4ccb19-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5b60f69-9404-49bd-a072-d04f9b4ccb19-workload-certs\") pod \"f5b60f69-9404-49bd-a072-d04f9b4ccb19\" (UID: \"f5b60f69-9404-49bd-a072-d04f9b4ccb19\") " Nov 03 21:08:15.773960 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.773891 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5b60f69-9404-49bd-a072-d04f9b4ccb19-istio-data" pod "f5b60f69-9404-49bd-a072-d04f9b4ccb19" (UID: "f5b60f69-9404-49bd-a072-d04f9b4ccb19"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:08:15.773960 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.773925 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5b60f69-9404-49bd-a072-d04f9b4ccb19-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b60f69-9404-49bd-a072-d04f9b4ccb19-workload-socket\") pod \"f5b60f69-9404-49bd-a072-d04f9b4ccb19\" (UID: \"f5b60f69-9404-49bd-a072-d04f9b4ccb19\") " Nov 03 21:08:15.774074 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.773990 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5b60f69-9404-49bd-a072-d04f9b4ccb19-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b60f69-9404-49bd-a072-d04f9b4ccb19-credential-socket\") pod \"f5b60f69-9404-49bd-a072-d04f9b4ccb19\" (UID: \"f5b60f69-9404-49bd-a072-d04f9b4ccb19\") " Nov 03 21:08:15.774074 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.774047 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5b60f69-9404-49bd-a072-d04f9b4ccb19-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5b60f69-9404-49bd-a072-d04f9b4ccb19-istiod-ca-cert\") pod \"f5b60f69-9404-49bd-a072-d04f9b4ccb19\" (UID: \"f5b60f69-9404-49bd-a072-d04f9b4ccb19\") " Nov 03 21:08:15.774418 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.774352 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5b60f69-9404-49bd-a072-d04f9b4ccb19-workload-socket" pod "f5b60f69-9404-49bd-a072-d04f9b4ccb19" (UID: "f5b60f69-9404-49bd-a072-d04f9b4ccb19"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:08:15.774418 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.774380 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5b60f69-9404-49bd-a072-d04f9b4ccb19-workload-certs" pod "f5b60f69-9404-49bd-a072-d04f9b4ccb19" (UID: "f5b60f69-9404-49bd-a072-d04f9b4ccb19"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:08:15.774550 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.774478 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5b60f69-9404-49bd-a072-d04f9b4ccb19-credential-socket" pod "f5b60f69-9404-49bd-a072-d04f9b4ccb19" (UID: "f5b60f69-9404-49bd-a072-d04f9b4ccb19"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:08:15.774823 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.774796 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5b60f69-9404-49bd-a072-d04f9b4ccb19-istiod-ca-cert" pod "f5b60f69-9404-49bd-a072-d04f9b4ccb19" (UID: "f5b60f69-9404-49bd-a072-d04f9b4ccb19"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:08:15.775194 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.775171 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5b60f69-9404-49bd-a072-d04f9b4ccb19-istio-data\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:08:15.775280 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.775199 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5b60f69-9404-49bd-a072-d04f9b4ccb19-workload-certs\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:08:15.775280 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.775215 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b60f69-9404-49bd-a072-d04f9b4ccb19-workload-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:08:15.775280 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.775229 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b60f69-9404-49bd-a072-d04f9b4ccb19-credential-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:08:15.775280 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.775244 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5b60f69-9404-49bd-a072-d04f9b4ccb19-istiod-ca-cert\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:08:15.775931 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.775907 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5b60f69-9404-49bd-a072-d04f9b4ccb19-istio-token" pod "f5b60f69-9404-49bd-a072-d04f9b4ccb19" (UID: "f5b60f69-9404-49bd-a072-d04f9b4ccb19"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:08:15.776076 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.776014 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5b60f69-9404-49bd-a072-d04f9b4ccb19-istio-envoy" pod "f5b60f69-9404-49bd-a072-d04f9b4ccb19" (UID: "f5b60f69-9404-49bd-a072-d04f9b4ccb19"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:08:15.776132 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.776071 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5b60f69-9404-49bd-a072-d04f9b4ccb19-istio-podinfo" pod "f5b60f69-9404-49bd-a072-d04f9b4ccb19" (UID: "f5b60f69-9404-49bd-a072-d04f9b4ccb19"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:08:15.776132 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.776076 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5b60f69-9404-49bd-a072-d04f9b4ccb19-kube-api-access-cjg42" pod "f5b60f69-9404-49bd-a072-d04f9b4ccb19" (UID: "f5b60f69-9404-49bd-a072-d04f9b4ccb19"). InnerVolumeSpecName "kube-api-access-cjg42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:08:15.877086 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.876982 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5b60f69-9404-49bd-a072-d04f9b4ccb19-istio-envoy\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:08:15.877086 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.877038 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5b60f69-9404-49bd-a072-d04f9b4ccb19-istio-token\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:08:15.877086 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.877052 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5b60f69-9404-49bd-a072-d04f9b4ccb19-istio-podinfo\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:08:15.877086 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.877065 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjg42\" (UniqueName: \"kubernetes.io/projected/f5b60f69-9404-49bd-a072-d04f9b4ccb19-kube-api-access-cjg42\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:08:15.902232 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.902197 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz"] Nov 03 21:08:15.906559 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:15.906536 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e3917193-openshift-default-6bf655bbd9-646wz"] Nov 03 21:08:16.157045 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:16.156984 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz"] Nov 03 21:08:16.164106 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:16.164077 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-5c2qz"] Nov 03 21:08:16.328364 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:08:16.328327 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:08:16.527150 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:16.527062 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk"] Nov 03 21:08:16.527377 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:16.527352 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk" podUID="d35870fa-5e32-4297-ba1b-4f6526cf4bfd" containerName="istio-proxy" containerID="cri-o://8acdbae0a25b9852b174ac8f53290dbcb0d7c8e980d968f171d0eed79d380641" gracePeriod=30 Nov 03 21:08:16.903074 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:16.903035 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14573b6a-c19e-42fe-85c6-f7ffa46db476" path="/var/lib/kubelet/pods/14573b6a-c19e-42fe-85c6-f7ffa46db476/volumes" Nov 03 21:08:16.903823 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:16.903795 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95dd3f17-e145-49ec-8fe2-6148c3f07c80" path="/var/lib/kubelet/pods/95dd3f17-e145-49ec-8fe2-6148c3f07c80/volumes" Nov 03 21:08:16.904485 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:16.904409 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5b60f69-9404-49bd-a072-d04f9b4ccb19" path="/var/lib/kubelet/pods/f5b60f69-9404-49bd-a072-d04f9b4ccb19/volumes" Nov 03 21:08:17.301686 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:17.301602 4305 patch_prober.go:29] interesting pod/e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:08:17.301686 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:17.301660 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk" podUID="d35870fa-5e32-4297-ba1b-4f6526cf4bfd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:08:19.102027 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:19.101992 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-89bb58-z292l"] Nov 03 21:08:19.102778 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:19.102760 4305 cpu_manager.go:414] "containerMap: removing container" podUID="b3ff046c-42eb-40ce-b555-d069fc6a8567" containerName="payload-pre-processing" Nov 03 21:08:19.102838 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:19.102779 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3ff046c-42eb-40ce-b555-d069fc6a8567" containerName="payload-pre-processing" Nov 03 21:08:19.102838 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:19.102793 4305 cpu_manager.go:414] "containerMap: removing container" podUID="95dd3f17-e145-49ec-8fe2-6148c3f07c80" containerName="istio-proxy" Nov 03 21:08:19.102838 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:19.102798 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95dd3f17-e145-49ec-8fe2-6148c3f07c80" containerName="istio-proxy" Nov 03 21:08:19.102838 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:19.102812 4305 cpu_manager.go:414] "containerMap: removing container" podUID="f5b60f69-9404-49bd-a072-d04f9b4ccb19" containerName="istio-proxy" Nov 03 21:08:19.102838 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:19.102816 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5b60f69-9404-49bd-a072-d04f9b4ccb19" containerName="istio-proxy" Nov 03 21:08:19.102838 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:19.102837 4305 cpu_manager.go:414] "containerMap: removing container" podUID="b6b1772b-0599-4077-a0df-c21e82e80d97" containerName="revoke-keys" Nov 03 21:08:19.103025 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:19.102842 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6b1772b-0599-4077-a0df-c21e82e80d97" containerName="revoke-keys" Nov 03 21:08:19.103025 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:19.102854 4305 cpu_manager.go:414] "containerMap: removing container" podUID="e37f3c67-e93b-4933-b01e-22a99ef52532" containerName="payload-processing" Nov 03 21:08:19.103025 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:19.102858 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e37f3c67-e93b-4933-b01e-22a99ef52532" containerName="payload-processing" Nov 03 21:08:19.103025 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:19.102873 4305 cpu_manager.go:414] "containerMap: removing container" podUID="2366aada-a95b-404a-b5a7-454a1670c687" containerName="istio-proxy" Nov 03 21:08:19.103025 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:19.102878 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2366aada-a95b-404a-b5a7-454a1670c687" containerName="istio-proxy" Nov 03 21:08:19.103025 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:19.102980 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6b1772b-0599-4077-a0df-c21e82e80d97" containerName="revoke-keys" Nov 03 21:08:19.103025 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:19.102991 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="e37f3c67-e93b-4933-b01e-22a99ef52532" containerName="payload-processing" Nov 03 21:08:19.103025 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:19.102999 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="2366aada-a95b-404a-b5a7-454a1670c687" containerName="istio-proxy" Nov 03 21:08:19.103025 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:19.103007 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3ff046c-42eb-40ce-b555-d069fc6a8567" containerName="payload-pre-processing" Nov 03 21:08:19.103025 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:19.103014 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="95dd3f17-e145-49ec-8fe2-6148c3f07c80" containerName="istio-proxy" Nov 03 21:08:19.103025 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:19.103022 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5b60f69-9404-49bd-a072-d04f9b4ccb19" containerName="istio-proxy" Nov 03 21:08:19.103349 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:19.103030 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="95dd3f17-e145-49ec-8fe2-6148c3f07c80" containerName="istio-proxy" Nov 03 21:08:19.107923 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:19.107895 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-89bb58-z292l" Nov 03 21:08:19.190212 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:19.190166 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8563a9a0-3bdf-43cb-96f1-3659a04f74d6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-89bb58-z292l\" (UID: \"8563a9a0-3bdf-43cb-96f1-3659a04f74d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-89bb58-z292l" Nov 03 21:08:19.190411 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:19.190353 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsldd\" (UniqueName: \"kubernetes.io/projected/8563a9a0-3bdf-43cb-96f1-3659a04f74d6-kube-api-access-bsldd\") pod \"maas-api-revoke-keys-e2e-worker-w3-89bb58-z292l\" (UID: \"8563a9a0-3bdf-43cb-96f1-3659a04f74d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-89bb58-z292l" Nov 03 21:08:19.190411 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:19.190394 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8563a9a0-3bdf-43cb-96f1-3659a04f74d6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-89bb58-z292l\" (UID: \"8563a9a0-3bdf-43cb-96f1-3659a04f74d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-89bb58-z292l" Nov 03 21:08:19.210130 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:19.210093 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:08:19.210130 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:19.210093 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77vtd\"" Nov 03 21:08:19.210425 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:19.210107 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:08:19.210553 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:19.210109 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:08:19.217082 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:19.217053 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-89bb58-z292l"] Nov 03 21:08:19.292100 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:19.292058 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8563a9a0-3bdf-43cb-96f1-3659a04f74d6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-89bb58-z292l\" (UID: \"8563a9a0-3bdf-43cb-96f1-3659a04f74d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-89bb58-z292l" Nov 03 21:08:19.292306 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:19.292199 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsldd\" (UniqueName: \"kubernetes.io/projected/8563a9a0-3bdf-43cb-96f1-3659a04f74d6-kube-api-access-bsldd\") pod \"maas-api-revoke-keys-e2e-worker-w3-89bb58-z292l\" (UID: \"8563a9a0-3bdf-43cb-96f1-3659a04f74d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-89bb58-z292l" Nov 03 21:08:19.292306 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:19.292232 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8563a9a0-3bdf-43cb-96f1-3659a04f74d6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-89bb58-z292l\" (UID: \"8563a9a0-3bdf-43cb-96f1-3659a04f74d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-89bb58-z292l" Nov 03 21:08:19.292795 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:19.292770 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8563a9a0-3bdf-43cb-96f1-3659a04f74d6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-89bb58-z292l\" (UID: \"8563a9a0-3bdf-43cb-96f1-3659a04f74d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-89bb58-z292l" Nov 03 21:08:19.294979 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:19.294956 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8563a9a0-3bdf-43cb-96f1-3659a04f74d6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-89bb58-z292l\" (UID: \"8563a9a0-3bdf-43cb-96f1-3659a04f74d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-89bb58-z292l" Nov 03 21:08:19.354228 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:19.354145 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsldd\" (UniqueName: \"kubernetes.io/projected/8563a9a0-3bdf-43cb-96f1-3659a04f74d6-kube-api-access-bsldd\") pod \"maas-api-revoke-keys-e2e-worker-w3-89bb58-z292l\" (UID: \"8563a9a0-3bdf-43cb-96f1-3659a04f74d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-89bb58-z292l" Nov 03 21:08:19.423134 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:19.423096 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-89bb58-z292l" Nov 03 21:08:19.810543 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:19.810510 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-89bb58-z292l"] Nov 03 21:08:19.811145 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:08:19.811115 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8563a9a0_3bdf_43cb_96f1_3659a04f74d6.slice/crio-9a933ea88f5f4c86759c6725e14cc2b3bf98f308343a3b88fa4eeee12e5385a2 WatchSource:0}: Error finding container 9a933ea88f5f4c86759c6725e14cc2b3bf98f308343a3b88fa4eeee12e5385a2: Status 404 returned error can't find the container with id 9a933ea88f5f4c86759c6725e14cc2b3bf98f308343a3b88fa4eeee12e5385a2 Nov 03 21:08:20.874276 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:20.874171 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-89bb58-z292l" event={"ID":"8563a9a0-3bdf-43cb-96f1-3659a04f74d6","Type":"ContainerStarted","Data":"0031641bfc57cf20bfe768a2d6924d674e775b54841e550dba97573cee0e0c95"} Nov 03 21:08:20.874852 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:20.874827 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-89bb58-z292l" event={"ID":"8563a9a0-3bdf-43cb-96f1-3659a04f74d6","Type":"ContainerStarted","Data":"9a933ea88f5f4c86759c6725e14cc2b3bf98f308343a3b88fa4eeee12e5385a2"} Nov 03 21:08:21.016361 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:21.016302 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-89bb58-z292l" podStartSLOduration=2.016286715 podStartE2EDuration="2.016286715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:08:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:08:21.014193358 +0000 UTC m=+1736.239513150" watchObservedRunningTime="2026-11-03 21:08:21.016286715 +0000 UTC m=+1736.241606519" Nov 03 21:08:21.838330 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:21.838299 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk" Nov 03 21:08:21.866161 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:21.866128 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d35870fa-5e32-4297-ba1b-4f6526cf4bfd-kube-api-access-scq7g\" (UniqueName: \"kubernetes.io/projected/d35870fa-5e32-4297-ba1b-4f6526cf4bfd-kube-api-access-scq7g\") pod \"d35870fa-5e32-4297-ba1b-4f6526cf4bfd\" (UID: \"d35870fa-5e32-4297-ba1b-4f6526cf4bfd\") " Nov 03 21:08:21.866349 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:21.866241 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d35870fa-5e32-4297-ba1b-4f6526cf4bfd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d35870fa-5e32-4297-ba1b-4f6526cf4bfd-workload-certs\") pod \"d35870fa-5e32-4297-ba1b-4f6526cf4bfd\" (UID: \"d35870fa-5e32-4297-ba1b-4f6526cf4bfd\") " Nov 03 21:08:21.866397 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:21.866375 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d35870fa-5e32-4297-ba1b-4f6526cf4bfd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d35870fa-5e32-4297-ba1b-4f6526cf4bfd-workload-socket\") pod \"d35870fa-5e32-4297-ba1b-4f6526cf4bfd\" (UID: \"d35870fa-5e32-4297-ba1b-4f6526cf4bfd\") " Nov 03 21:08:21.866525 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:21.866509 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d35870fa-5e32-4297-ba1b-4f6526cf4bfd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d35870fa-5e32-4297-ba1b-4f6526cf4bfd-istiod-ca-cert\") pod \"d35870fa-5e32-4297-ba1b-4f6526cf4bfd\" (UID: \"d35870fa-5e32-4297-ba1b-4f6526cf4bfd\") " Nov 03 21:08:21.866597 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:21.866582 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d35870fa-5e32-4297-ba1b-4f6526cf4bfd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d35870fa-5e32-4297-ba1b-4f6526cf4bfd-istio-podinfo\") pod \"d35870fa-5e32-4297-ba1b-4f6526cf4bfd\" (UID: \"d35870fa-5e32-4297-ba1b-4f6526cf4bfd\") " Nov 03 21:08:21.866636 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:21.866584 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d35870fa-5e32-4297-ba1b-4f6526cf4bfd-workload-certs" pod "d35870fa-5e32-4297-ba1b-4f6526cf4bfd" (UID: "d35870fa-5e32-4297-ba1b-4f6526cf4bfd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:08:21.866636 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:21.866603 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d35870fa-5e32-4297-ba1b-4f6526cf4bfd-workload-socket" pod "d35870fa-5e32-4297-ba1b-4f6526cf4bfd" (UID: "d35870fa-5e32-4297-ba1b-4f6526cf4bfd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:08:21.866636 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:21.866620 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d35870fa-5e32-4297-ba1b-4f6526cf4bfd-istio-token\" (UniqueName: \"kubernetes.io/projected/d35870fa-5e32-4297-ba1b-4f6526cf4bfd-istio-token\") pod \"d35870fa-5e32-4297-ba1b-4f6526cf4bfd\" (UID: \"d35870fa-5e32-4297-ba1b-4f6526cf4bfd\") " Nov 03 21:08:21.866778 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:21.866672 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d35870fa-5e32-4297-ba1b-4f6526cf4bfd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d35870fa-5e32-4297-ba1b-4f6526cf4bfd-istio-data\") pod \"d35870fa-5e32-4297-ba1b-4f6526cf4bfd\" (UID: \"d35870fa-5e32-4297-ba1b-4f6526cf4bfd\") " Nov 03 21:08:21.866778 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:21.866699 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d35870fa-5e32-4297-ba1b-4f6526cf4bfd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d35870fa-5e32-4297-ba1b-4f6526cf4bfd-istio-envoy\") pod \"d35870fa-5e32-4297-ba1b-4f6526cf4bfd\" (UID: \"d35870fa-5e32-4297-ba1b-4f6526cf4bfd\") " Nov 03 21:08:21.866778 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:21.866731 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d35870fa-5e32-4297-ba1b-4f6526cf4bfd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d35870fa-5e32-4297-ba1b-4f6526cf4bfd-credential-socket\") pod \"d35870fa-5e32-4297-ba1b-4f6526cf4bfd\" (UID: \"d35870fa-5e32-4297-ba1b-4f6526cf4bfd\") " Nov 03 21:08:21.867050 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:21.866990 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d35870fa-5e32-4297-ba1b-4f6526cf4bfd-istio-data" pod "d35870fa-5e32-4297-ba1b-4f6526cf4bfd" (UID: "d35870fa-5e32-4297-ba1b-4f6526cf4bfd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:08:21.867050 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:21.867018 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d35870fa-5e32-4297-ba1b-4f6526cf4bfd-istiod-ca-cert" pod "d35870fa-5e32-4297-ba1b-4f6526cf4bfd" (UID: "d35870fa-5e32-4297-ba1b-4f6526cf4bfd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:08:21.867246 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:21.867188 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d35870fa-5e32-4297-ba1b-4f6526cf4bfd-credential-socket" pod "d35870fa-5e32-4297-ba1b-4f6526cf4bfd" (UID: "d35870fa-5e32-4297-ba1b-4f6526cf4bfd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:08:21.867809 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:21.867787 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d35870fa-5e32-4297-ba1b-4f6526cf4bfd-workload-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:08:21.867860 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:21.867819 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d35870fa-5e32-4297-ba1b-4f6526cf4bfd-istiod-ca-cert\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:08:21.867860 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:21.867835 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d35870fa-5e32-4297-ba1b-4f6526cf4bfd-istio-data\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:08:21.867860 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:21.867848 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d35870fa-5e32-4297-ba1b-4f6526cf4bfd-credential-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:08:21.867995 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:21.867861 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d35870fa-5e32-4297-ba1b-4f6526cf4bfd-workload-certs\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:08:21.869043 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:21.869022 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d35870fa-5e32-4297-ba1b-4f6526cf4bfd-istio-podinfo" pod "d35870fa-5e32-4297-ba1b-4f6526cf4bfd" (UID: "d35870fa-5e32-4297-ba1b-4f6526cf4bfd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:08:21.869142 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:21.869057 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d35870fa-5e32-4297-ba1b-4f6526cf4bfd-istio-envoy" pod "d35870fa-5e32-4297-ba1b-4f6526cf4bfd" (UID: "d35870fa-5e32-4297-ba1b-4f6526cf4bfd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:08:21.869264 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:21.869247 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d35870fa-5e32-4297-ba1b-4f6526cf4bfd-istio-token" pod "d35870fa-5e32-4297-ba1b-4f6526cf4bfd" (UID: "d35870fa-5e32-4297-ba1b-4f6526cf4bfd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:08:21.869324 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:21.869272 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d35870fa-5e32-4297-ba1b-4f6526cf4bfd-kube-api-access-scq7g" pod "d35870fa-5e32-4297-ba1b-4f6526cf4bfd" (UID: "d35870fa-5e32-4297-ba1b-4f6526cf4bfd"). InnerVolumeSpecName "kube-api-access-scq7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:08:21.884569 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:21.884538 4305 generic.go:356] "Generic (PLEG): container finished" podID="d35870fa-5e32-4297-ba1b-4f6526cf4bfd" containerID="8acdbae0a25b9852b174ac8f53290dbcb0d7c8e980d968f171d0eed79d380641" exitCode=0 Nov 03 21:08:21.885047 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:21.884597 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk" Nov 03 21:08:21.885047 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:21.884620 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk" event={"ID":"d35870fa-5e32-4297-ba1b-4f6526cf4bfd","Type":"ContainerDied","Data":"8acdbae0a25b9852b174ac8f53290dbcb0d7c8e980d968f171d0eed79d380641"} Nov 03 21:08:21.885047 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:21.884682 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk" event={"ID":"d35870fa-5e32-4297-ba1b-4f6526cf4bfd","Type":"ContainerDied","Data":"4b292bd433e26333a32ed969be0968a5974e46439f08cb34c53e93e06a7e5e5a"} Nov 03 21:08:21.885047 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:21.884705 4305 scope.go:122] "RemoveContainer" containerID="8acdbae0a25b9852b174ac8f53290dbcb0d7c8e980d968f171d0eed79d380641" Nov 03 21:08:21.886840 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:21.886816 4305 generic.go:356] "Generic (PLEG): container finished" podID="8563a9a0-3bdf-43cb-96f1-3659a04f74d6" containerID="0031641bfc57cf20bfe768a2d6924d674e775b54841e550dba97573cee0e0c95" exitCode=0 Nov 03 21:08:21.886958 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:21.886888 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-89bb58-z292l" event={"ID":"8563a9a0-3bdf-43cb-96f1-3659a04f74d6","Type":"ContainerDied","Data":"0031641bfc57cf20bfe768a2d6924d674e775b54841e550dba97573cee0e0c95"} Nov 03 21:08:21.904224 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:21.904201 4305 scope.go:122] "RemoveContainer" containerID="74828063f3048ec2b674488e221b95d6107efb955c64ef034de43bef4eb62b9b" Nov 03 21:08:21.915295 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:21.915274 4305 scope.go:122] "RemoveContainer" containerID="8acdbae0a25b9852b174ac8f53290dbcb0d7c8e980d968f171d0eed79d380641" Nov 03 21:08:21.915660 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:08:21.915639 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8acdbae0a25b9852b174ac8f53290dbcb0d7c8e980d968f171d0eed79d380641\": container with ID starting with 8acdbae0a25b9852b174ac8f53290dbcb0d7c8e980d968f171d0eed79d380641 not found: ID does not exist" containerID="8acdbae0a25b9852b174ac8f53290dbcb0d7c8e980d968f171d0eed79d380641" Nov 03 21:08:21.915752 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:21.915672 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8acdbae0a25b9852b174ac8f53290dbcb0d7c8e980d968f171d0eed79d380641"} err="failed to get container status \"8acdbae0a25b9852b174ac8f53290dbcb0d7c8e980d968f171d0eed79d380641\": rpc error: code = NotFound desc = could not find container \"8acdbae0a25b9852b174ac8f53290dbcb0d7c8e980d968f171d0eed79d380641\": container with ID starting with 8acdbae0a25b9852b174ac8f53290dbcb0d7c8e980d968f171d0eed79d380641 not found: ID does not exist" Nov 03 21:08:21.915752 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:21.915696 4305 scope.go:122] "RemoveContainer" containerID="74828063f3048ec2b674488e221b95d6107efb955c64ef034de43bef4eb62b9b" Nov 03 21:08:21.915958 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:08:21.915923 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74828063f3048ec2b674488e221b95d6107efb955c64ef034de43bef4eb62b9b\": container with ID starting with 74828063f3048ec2b674488e221b95d6107efb955c64ef034de43bef4eb62b9b not found: ID does not exist" containerID="74828063f3048ec2b674488e221b95d6107efb955c64ef034de43bef4eb62b9b" Nov 03 21:08:21.916006 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:21.915971 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74828063f3048ec2b674488e221b95d6107efb955c64ef034de43bef4eb62b9b"} err="failed to get container status \"74828063f3048ec2b674488e221b95d6107efb955c64ef034de43bef4eb62b9b\": rpc error: code = NotFound desc = could not find container \"74828063f3048ec2b674488e221b95d6107efb955c64ef034de43bef4eb62b9b\": container with ID starting with 74828063f3048ec2b674488e221b95d6107efb955c64ef034de43bef4eb62b9b not found: ID does not exist" Nov 03 21:08:21.970590 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:21.970557 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scq7g\" (UniqueName: \"kubernetes.io/projected/d35870fa-5e32-4297-ba1b-4f6526cf4bfd-kube-api-access-scq7g\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:08:21.970590 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:21.970594 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d35870fa-5e32-4297-ba1b-4f6526cf4bfd-istio-podinfo\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:08:21.970824 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:21.970607 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d35870fa-5e32-4297-ba1b-4f6526cf4bfd-istio-token\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:08:21.970824 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:21.970616 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d35870fa-5e32-4297-ba1b-4f6526cf4bfd-istio-envoy\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:08:22.112792 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:22.112717 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04f5216c-4shmx"] Nov 03 21:08:22.116774 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:22.116744 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-04f5216c-4shmx"] Nov 03 21:08:22.133358 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:22.133328 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk"] Nov 03 21:08:22.137504 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:22.137476 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-e41159db-openshift-default-7976dbf4c-chdsk"] Nov 03 21:08:22.994797 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:22.994758 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="075cc1fc-27d6-42a9-a068-1240584106bc" path="/var/lib/kubelet/pods/075cc1fc-27d6-42a9-a068-1240584106bc/volumes" Nov 03 21:08:22.995709 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:22.995679 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d35870fa-5e32-4297-ba1b-4f6526cf4bfd" path="/var/lib/kubelet/pods/d35870fa-5e32-4297-ba1b-4f6526cf4bfd/volumes" Nov 03 21:08:23.168332 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:23.168299 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-89bb58-z292l" Nov 03 21:08:23.200296 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:23.200267 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8563a9a0-3bdf-43cb-96f1-3659a04f74d6-kube-api-access-bsldd\" (UniqueName: \"kubernetes.io/projected/8563a9a0-3bdf-43cb-96f1-3659a04f74d6-kube-api-access-bsldd\") pod \"8563a9a0-3bdf-43cb-96f1-3659a04f74d6\" (UID: \"8563a9a0-3bdf-43cb-96f1-3659a04f74d6\") " Nov 03 21:08:23.200490 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:23.200345 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8563a9a0-3bdf-43cb-96f1-3659a04f74d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8563a9a0-3bdf-43cb-96f1-3659a04f74d6-maas-api-xks-ca\") pod \"8563a9a0-3bdf-43cb-96f1-3659a04f74d6\" (UID: \"8563a9a0-3bdf-43cb-96f1-3659a04f74d6\") " Nov 03 21:08:23.200545 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:23.200506 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8563a9a0-3bdf-43cb-96f1-3659a04f74d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8563a9a0-3bdf-43cb-96f1-3659a04f74d6-maas-api-openshift-ca\") pod \"8563a9a0-3bdf-43cb-96f1-3659a04f74d6\" (UID: \"8563a9a0-3bdf-43cb-96f1-3659a04f74d6\") " Nov 03 21:08:23.200922 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:23.200892 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8563a9a0-3bdf-43cb-96f1-3659a04f74d6-maas-api-openshift-ca" pod "8563a9a0-3bdf-43cb-96f1-3659a04f74d6" (UID: "8563a9a0-3bdf-43cb-96f1-3659a04f74d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:08:23.201126 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:23.201077 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8563a9a0-3bdf-43cb-96f1-3659a04f74d6-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:08:23.202660 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:23.202633 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8563a9a0-3bdf-43cb-96f1-3659a04f74d6-maas-api-xks-ca" pod "8563a9a0-3bdf-43cb-96f1-3659a04f74d6" (UID: "8563a9a0-3bdf-43cb-96f1-3659a04f74d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:08:23.202765 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:23.202716 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8563a9a0-3bdf-43cb-96f1-3659a04f74d6-kube-api-access-bsldd" pod "8563a9a0-3bdf-43cb-96f1-3659a04f74d6" (UID: "8563a9a0-3bdf-43cb-96f1-3659a04f74d6"). InnerVolumeSpecName "kube-api-access-bsldd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:08:23.303113 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:23.303029 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsldd\" (UniqueName: \"kubernetes.io/projected/8563a9a0-3bdf-43cb-96f1-3659a04f74d6-kube-api-access-bsldd\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:08:23.303113 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:23.303067 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8563a9a0-3bdf-43cb-96f1-3659a04f74d6-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:08:23.442320 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:23.442285 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj"] Nov 03 21:08:23.450698 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:23.450668 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g"] Nov 03 21:08:23.999039 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:23.999011 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-89bb58-z292l" Nov 03 21:08:23.999459 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:23.999041 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-89bb58-z292l" event={"ID":"8563a9a0-3bdf-43cb-96f1-3659a04f74d6","Type":"ContainerDied","Data":"9a933ea88f5f4c86759c6725e14cc2b3bf98f308343a3b88fa4eeee12e5385a2"} Nov 03 21:08:23.999459 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:23.999092 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a933ea88f5f4c86759c6725e14cc2b3bf98f308343a3b88fa4eeee12e5385a2" Nov 03 21:08:30.965134 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:30.965099 4305 scope.go:122] "RemoveContainer" containerID="63b7bd9c85ae7ec4a51da08ed59af05a60277e7f2350fbdc587cfc2fbc903b7f" Nov 03 21:08:30.977062 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:30.977038 4305 scope.go:122] "RemoveContainer" containerID="738c045b398f6baa821200ef859f3633882fc1656307ccd12549521f6f000bc3" Nov 03 21:08:30.987068 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:30.987047 4305 scope.go:122] "RemoveContainer" containerID="13c25083621c3ab3656bb562ebc0c9d3223d53988839c19668692f73b94e7248" Nov 03 21:08:30.997112 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:30.997093 4305 scope.go:122] "RemoveContainer" containerID="55383d33fd02610b7c73156a75c26bf7a11e1eca4424338ab8bf9f652fbc6116" Nov 03 21:08:31.007464 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:31.007424 4305 scope.go:122] "RemoveContainer" containerID="69279a13c80d80e4c6bed49347dd4d876bc6b6e3c7ab55b2ddf43baea8545e5f" Nov 03 21:08:31.018279 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:31.018258 4305 scope.go:122] "RemoveContainer" containerID="753e3ed999f626f0f8d4f41f1ef6bf2cf6c09ca205110f86110bd73159959fcf" Nov 03 21:08:31.028251 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:31.028229 4305 scope.go:122] "RemoveContainer" containerID="06b88d529ebf9d878bb385fec6c6c36c3160cdc9588cdbfa25afd7e77a98a776" Nov 03 21:08:31.038560 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:31.038533 4305 scope.go:122] "RemoveContainer" containerID="2dad96b48251c930abeb21eaf534e4fe5f98fde708cd7c7c3d83192b2f4c1185" Nov 03 21:08:31.049517 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:31.049494 4305 scope.go:122] "RemoveContainer" containerID="e68d0b79d6f99eeaee550444c8a88915d3c3f18fd152d092f72d44c02220107a" Nov 03 21:08:31.063789 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:31.063758 4305 scope.go:122] "RemoveContainer" containerID="6c805e225e5f7e1244c5b662867c7baccd701cf49678c9cb1c5fab6f4f17c742" Nov 03 21:08:31.073995 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:31.073976 4305 scope.go:122] "RemoveContainer" containerID="16099414780cfc6f0e43de40711a4c98c7175da4ec768980d02847e39495412b" Nov 03 21:08:38.466184 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:38.466103 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" podUID="8793f095-97b1-4e0a-8382-b541f90a0de2" containerName="main" containerID="cri-o://757468dfac9960a8b7021a8d33be8a19cc19b93f4d0952d92a933ed78d858d01" gracePeriod=45 Nov 03 21:08:38.475845 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:38.475787 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" podUID="7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1" containerName="main" containerID="cri-o://af324f45577a5dbeb6e03ba7f663ed039a97cc397d4d4995a69ac382110096ee" gracePeriod=45 Nov 03 21:08:38.840502 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:38.823025 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" podUID="8793f095-97b1-4e0a-8382-b541f90a0de2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": dial tcp 10.134.0.79:8000: connect: connection refused" Nov 03 21:08:38.840502 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:38.827760 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" podUID="7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": dial tcp 10.134.0.60:8000: connect: connection refused" Nov 03 21:08:39.027524 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:39.027426 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:08:39.157879 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:39.157846 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" Nov 03 21:08:39.163483 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:39.163403 4305 generic.go:356] "Generic (PLEG): container finished" podID="8793f095-97b1-4e0a-8382-b541f90a0de2" containerID="757468dfac9960a8b7021a8d33be8a19cc19b93f4d0952d92a933ed78d858d01" exitCode=0 Nov 03 21:08:39.163820 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:39.163704 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" event={"ID":"8793f095-97b1-4e0a-8382-b541f90a0de2","Type":"ContainerDied","Data":"757468dfac9960a8b7021a8d33be8a19cc19b93f4d0952d92a933ed78d858d01"} Nov 03 21:08:39.166140 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:39.165504 4305 generic.go:356] "Generic (PLEG): container finished" podID="7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1" containerID="af324f45577a5dbeb6e03ba7f663ed039a97cc397d4d4995a69ac382110096ee" exitCode=0 Nov 03 21:08:39.166140 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:39.165576 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" event={"ID":"7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1","Type":"ContainerDied","Data":"af324f45577a5dbeb6e03ba7f663ed039a97cc397d4d4995a69ac382110096ee"} Nov 03 21:08:39.166140 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:39.165610 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" event={"ID":"7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1","Type":"ContainerDied","Data":"6262bc5f41af83ab2e2edd57e7f866a9da46f7705fddb8fda0aafa65a2414423"} Nov 03 21:08:39.166140 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:39.165633 4305 scope.go:122] "RemoveContainer" containerID="af324f45577a5dbeb6e03ba7f663ed039a97cc397d4d4995a69ac382110096ee" Nov 03 21:08:39.166140 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:39.165808 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g" Nov 03 21:08:39.216900 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:39.209939 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" Nov 03 21:08:39.224702 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:39.224665 4305 scope.go:122] "RemoveContainer" containerID="af324f45577a5dbeb6e03ba7f663ed039a97cc397d4d4995a69ac382110096ee" Nov 03 21:08:39.225959 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:39.225168 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1-kube-api-access-c6hpg\" (UniqueName: \"kubernetes.io/projected/7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1-kube-api-access-c6hpg\") pod \"7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1\" (UID: \"7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1\") " Nov 03 21:08:39.225959 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:08:39.225169 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af324f45577a5dbeb6e03ba7f663ed039a97cc397d4d4995a69ac382110096ee\": container with ID starting with af324f45577a5dbeb6e03ba7f663ed039a97cc397d4d4995a69ac382110096ee not found: ID does not exist" containerID="af324f45577a5dbeb6e03ba7f663ed039a97cc397d4d4995a69ac382110096ee" Nov 03 21:08:39.225959 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:39.225230 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1-home\" (UniqueName: \"kubernetes.io/empty-dir/7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1-home\") pod \"7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1\" (UID: \"7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1\") " Nov 03 21:08:39.225959 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:39.225238 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af324f45577a5dbeb6e03ba7f663ed039a97cc397d4d4995a69ac382110096ee"} err="failed to get container status \"af324f45577a5dbeb6e03ba7f663ed039a97cc397d4d4995a69ac382110096ee\": rpc error: code = NotFound desc = could not find container \"af324f45577a5dbeb6e03ba7f663ed039a97cc397d4d4995a69ac382110096ee\": container with ID starting with af324f45577a5dbeb6e03ba7f663ed039a97cc397d4d4995a69ac382110096ee not found: ID does not exist" Nov 03 21:08:39.225959 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:39.225318 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1-dshm\") pod \"7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1\" (UID: \"7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1\") " Nov 03 21:08:39.225959 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:39.225577 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1-tls-certs\" (UniqueName: \"kubernetes.io/secret/7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1-tls-certs\") pod \"7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1\" (UID: \"7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1\") " Nov 03 21:08:39.225959 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:39.225676 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1-tmp-dir\") pod \"7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1\" (UID: \"7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1\") " Nov 03 21:08:39.225959 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:39.225773 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1-home" pod "7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1" (UID: "7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:08:39.225959 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:39.225793 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1-model-cache\") pod \"7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1\" (UID: \"7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1\") " Nov 03 21:08:39.227482 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:39.227254 4305 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1-home\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:08:39.227685 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:39.227640 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1-model-cache" pod "7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1" (UID: "7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:08:39.227876 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:39.227856 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1-tmp-dir" pod "7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1" (UID: "7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:08:39.229173 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:39.229141 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1-tls-certs" pod "7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1" (UID: "7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:08:39.231620 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:39.231587 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1-kube-api-access-c6hpg" pod "7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1" (UID: "7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1"). InnerVolumeSpecName "kube-api-access-c6hpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:08:39.231620 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:39.231614 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1-dshm" pod "7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1" (UID: "7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:08:39.327892 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:39.327812 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8793f095-97b1-4e0a-8382-b541f90a0de2-kube-api-access-47bjp\" (UniqueName: \"kubernetes.io/projected/8793f095-97b1-4e0a-8382-b541f90a0de2-kube-api-access-47bjp\") pod \"8793f095-97b1-4e0a-8382-b541f90a0de2\" (UID: \"8793f095-97b1-4e0a-8382-b541f90a0de2\") " Nov 03 21:08:39.328059 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:39.327913 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8793f095-97b1-4e0a-8382-b541f90a0de2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8793f095-97b1-4e0a-8382-b541f90a0de2-tmp-dir\") pod \"8793f095-97b1-4e0a-8382-b541f90a0de2\" (UID: \"8793f095-97b1-4e0a-8382-b541f90a0de2\") " Nov 03 21:08:39.328059 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:39.327972 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8793f095-97b1-4e0a-8382-b541f90a0de2-tls-certs\" (UniqueName: \"kubernetes.io/secret/8793f095-97b1-4e0a-8382-b541f90a0de2-tls-certs\") pod \"8793f095-97b1-4e0a-8382-b541f90a0de2\" (UID: \"8793f095-97b1-4e0a-8382-b541f90a0de2\") " Nov 03 21:08:39.328059 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:39.328051 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8793f095-97b1-4e0a-8382-b541f90a0de2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8793f095-97b1-4e0a-8382-b541f90a0de2-dshm\") pod \"8793f095-97b1-4e0a-8382-b541f90a0de2\" (UID: \"8793f095-97b1-4e0a-8382-b541f90a0de2\") " Nov 03 21:08:39.328186 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:39.328170 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8793f095-97b1-4e0a-8382-b541f90a0de2-home\" (UniqueName: \"kubernetes.io/empty-dir/8793f095-97b1-4e0a-8382-b541f90a0de2-home\") pod \"8793f095-97b1-4e0a-8382-b541f90a0de2\" (UID: \"8793f095-97b1-4e0a-8382-b541f90a0de2\") " Nov 03 21:08:39.328418 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:39.328399 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8793f095-97b1-4e0a-8382-b541f90a0de2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8793f095-97b1-4e0a-8382-b541f90a0de2-model-cache\") pod \"8793f095-97b1-4e0a-8382-b541f90a0de2\" (UID: \"8793f095-97b1-4e0a-8382-b541f90a0de2\") " Nov 03 21:08:39.329757 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:39.329731 4305 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1-tls-certs\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:08:39.329939 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:39.329915 4305 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1-tmp-dir\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:08:39.330052 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:39.329944 4305 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1-model-cache\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:08:39.330052 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:39.329961 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6hpg\" (UniqueName: \"kubernetes.io/projected/7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1-kube-api-access-c6hpg\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:08:39.330052 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:39.330018 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8793f095-97b1-4e0a-8382-b541f90a0de2-model-cache" pod "8793f095-97b1-4e0a-8382-b541f90a0de2" (UID: "8793f095-97b1-4e0a-8382-b541f90a0de2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:08:39.330218 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:39.330093 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8793f095-97b1-4e0a-8382-b541f90a0de2-tmp-dir" pod "8793f095-97b1-4e0a-8382-b541f90a0de2" (UID: "8793f095-97b1-4e0a-8382-b541f90a0de2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:08:39.330218 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:39.329979 4305 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1-dshm\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:08:39.330218 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:39.330143 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8793f095-97b1-4e0a-8382-b541f90a0de2-home" pod "8793f095-97b1-4e0a-8382-b541f90a0de2" (UID: "8793f095-97b1-4e0a-8382-b541f90a0de2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:08:39.332370 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:39.332326 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8793f095-97b1-4e0a-8382-b541f90a0de2-tls-certs" pod "8793f095-97b1-4e0a-8382-b541f90a0de2" (UID: "8793f095-97b1-4e0a-8382-b541f90a0de2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:08:39.334219 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:39.333212 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8793f095-97b1-4e0a-8382-b541f90a0de2-kube-api-access-47bjp" pod "8793f095-97b1-4e0a-8382-b541f90a0de2" (UID: "8793f095-97b1-4e0a-8382-b541f90a0de2"). InnerVolumeSpecName "kube-api-access-47bjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:08:39.335426 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:39.335398 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8793f095-97b1-4e0a-8382-b541f90a0de2-dshm" pod "8793f095-97b1-4e0a-8382-b541f90a0de2" (UID: "8793f095-97b1-4e0a-8382-b541f90a0de2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:08:39.431797 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:39.431761 4305 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8793f095-97b1-4e0a-8382-b541f90a0de2-home\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:08:39.431797 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:39.431795 4305 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8793f095-97b1-4e0a-8382-b541f90a0de2-model-cache\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:08:39.432030 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:39.431809 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47bjp\" (UniqueName: \"kubernetes.io/projected/8793f095-97b1-4e0a-8382-b541f90a0de2-kube-api-access-47bjp\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:08:39.432030 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:39.431823 4305 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8793f095-97b1-4e0a-8382-b541f90a0de2-tmp-dir\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:08:39.432030 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:39.431835 4305 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8793f095-97b1-4e0a-8382-b541f90a0de2-tls-certs\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:08:39.432030 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:39.431846 4305 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8793f095-97b1-4e0a-8382-b541f90a0de2-dshm\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:08:39.753914 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:39.753872 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g"] Nov 03 21:08:39.758142 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:39.758104 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-89bb58/premium-w3-89bb58-kserve-59cdd84977-dmf9g"] Nov 03 21:08:40.178822 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:40.178779 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" event={"ID":"8793f095-97b1-4e0a-8382-b541f90a0de2","Type":"ContainerDied","Data":"d8f79b1fd43b37216ba0c6131eeea5e5480b2b73e8db18e2be7500e0c94ce5d0"} Nov 03 21:08:40.179041 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:40.178847 4305 scope.go:122] "RemoveContainer" containerID="757468dfac9960a8b7021a8d33be8a19cc19b93f4d0952d92a933ed78d858d01" Nov 03 21:08:40.179041 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:40.178844 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj" Nov 03 21:08:40.375458 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:40.375169 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj"] Nov 03 21:08:40.410554 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:40.410506 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-89bb58/distinct-w3-89bb58-kserve-5cb775b97b-pvwsj"] Nov 03 21:08:40.935459 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:40.935095 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1" path="/var/lib/kubelet/pods/7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1/volumes" Nov 03 21:08:40.970818 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:40.968689 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8793f095-97b1-4e0a-8382-b541f90a0de2" path="/var/lib/kubelet/pods/8793f095-97b1-4e0a-8382-b541f90a0de2/volumes" Nov 03 21:08:42.890480 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:42.876592 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" podUID="80de7944-70f6-49e1-a0e8-4b82d5caad0d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:08:46.323772 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:08:46.323734 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:08:54.570025 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:54.569986 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x"] Nov 03 21:08:54.570554 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:54.570267 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x" podUID="1920ffc0-36e7-4d99-9410-444ad24afd9c" containerName="istio-proxy" containerID="cri-o://dfbf78fe82e05e2a90b38ad21dac6ac7e2793aa394447992c7651e802a95478d" gracePeriod=30 Nov 03 21:08:54.596426 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:54.596387 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d"] Nov 03 21:08:54.596669 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:54.596641 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d" podUID="d9103e89-52f2-4022-bd9b-dc554c82f479" containerName="istio-proxy" containerID="cri-o://d950b9777f3fe72d899dfe98f108d37bb059466e0e8b4c1b5216870ecd4bafcc" gracePeriod=30 Nov 03 21:08:54.602051 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:54.602011 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd"] Nov 03 21:08:54.602279 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:54.602256 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd" podUID="0ff154f8-a2c6-49a4-bbc8-f5026ec757bd" containerName="istio-proxy" containerID="cri-o://4d111a68c0e46f09480dfdfe969cb1010022c19b4dd169cff64e9ee661f2334c" gracePeriod=30 Nov 03 21:08:54.656194 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:54.656151 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh"] Nov 03 21:08:54.656482 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:54.656454 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh" podUID="17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1" containerName="istio-proxy" containerID="cri-o://ea7e5f654bccbde6382c9f018f3f2ec5f2bc78514d2af0f94c66559e0d9aae46" gracePeriod=30 Nov 03 21:08:54.689736 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:54.689694 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx"] Nov 03 21:08:54.690581 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:54.689990 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx" podUID="36f2d843-7d6a-4606-b6b3-6904a2b6cc29" containerName="istio-proxy" containerID="cri-o://8818f76d7f9e22d97b1005dad732e19ae4011a562bc75443701eec18a1f9726e" gracePeriod=30 Nov 03 21:08:54.703968 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:54.703883 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28"] Nov 03 21:08:54.704213 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:54.704181 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28" podUID="58829723-0628-457e-80b8-f81d15ecfe2e" containerName="istio-proxy" containerID="cri-o://bf7eed539704b28314d8666f8366e928a9a1acdf0bd6773de39b3c6c370f01a9" gracePeriod=30 Nov 03 21:08:54.709917 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:54.709883 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd"] Nov 03 21:08:54.710231 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:54.710198 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd" podUID="94bcfeb9-714d-4f25-8ce7-c6c4577ea837" containerName="istio-proxy" containerID="cri-o://b1008eb64b520f663b7e82c72eecf11ea5d8037cbd6c6f312cfe85e5a20e97b7" gracePeriod=30 Nov 03 21:08:58.119639 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:58.119598 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:08:58.120112 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:58.119668 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh" podUID="17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:08:58.437733 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:58.437697 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb6523bb-dd4qf"] Nov 03 21:08:58.438399 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:58.438369 4305 cpu_manager.go:414] "containerMap: removing container" podUID="d35870fa-5e32-4297-ba1b-4f6526cf4bfd" containerName="istio-proxy" Nov 03 21:08:58.438399 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:58.438388 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d35870fa-5e32-4297-ba1b-4f6526cf4bfd" containerName="istio-proxy" Nov 03 21:08:58.438399 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:58.438406 4305 cpu_manager.go:414] "containerMap: removing container" podUID="8563a9a0-3bdf-43cb-96f1-3659a04f74d6" containerName="revoke-keys" Nov 03 21:08:58.438680 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:58.438413 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8563a9a0-3bdf-43cb-96f1-3659a04f74d6" containerName="revoke-keys" Nov 03 21:08:58.438680 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:58.438448 4305 cpu_manager.go:414] "containerMap: removing container" podUID="8793f095-97b1-4e0a-8382-b541f90a0de2" containerName="main" Nov 03 21:08:58.438680 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:58.438456 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8793f095-97b1-4e0a-8382-b541f90a0de2" containerName="main" Nov 03 21:08:58.438680 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:58.438497 4305 cpu_manager.go:414] "containerMap: removing container" podUID="7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1" containerName="main" Nov 03 21:08:58.438680 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:58.438505 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1" containerName="main" Nov 03 21:08:58.438680 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:58.438533 4305 cpu_manager.go:414] "containerMap: removing container" podUID="95dd3f17-e145-49ec-8fe2-6148c3f07c80" containerName="istio-proxy" Nov 03 21:08:58.438680 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:58.438541 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95dd3f17-e145-49ec-8fe2-6148c3f07c80" containerName="istio-proxy" Nov 03 21:08:58.439019 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:58.438725 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="d35870fa-5e32-4297-ba1b-4f6526cf4bfd" containerName="istio-proxy" Nov 03 21:08:58.439019 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:58.438741 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f8e4e8b-bb78-4b3d-bc7d-2a35d9a05fb1" containerName="main" Nov 03 21:08:58.439019 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:58.438757 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="d35870fa-5e32-4297-ba1b-4f6526cf4bfd" containerName="istio-proxy" Nov 03 21:08:58.439019 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:58.438768 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="8563a9a0-3bdf-43cb-96f1-3659a04f74d6" containerName="revoke-keys" Nov 03 21:08:58.439019 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:58.438778 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="8793f095-97b1-4e0a-8382-b541f90a0de2" containerName="main" Nov 03 21:08:58.442593 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:58.442572 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb6523bb-dd4qf" Nov 03 21:08:58.481732 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:58.481693 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:08:58.481911 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:58.481767 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x" podUID="1920ffc0-36e7-4d99-9410-444ad24afd9c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:08:58.498492 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:58.498296 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-346e532f-lztv8"] Nov 03 21:08:58.499373 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:58.499333 4305 cpu_manager.go:414] "containerMap: removing container" podUID="d35870fa-5e32-4297-ba1b-4f6526cf4bfd" containerName="istio-proxy" Nov 03 21:08:58.499373 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:58.499358 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d35870fa-5e32-4297-ba1b-4f6526cf4bfd" containerName="istio-proxy" Nov 03 21:08:58.504097 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:58.504070 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-346e532f-lztv8" Nov 03 21:08:58.508296 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:58.508267 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb6523bb-dd4qf"] Nov 03 21:08:58.542799 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:58.542764 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:08:58.546089 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:58.546050 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:08:58.546089 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:58.546052 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:08:58.546287 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:58.546108 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77vtd\"" Nov 03 21:08:58.558135 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:58.558105 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fb6d58d-39c4-4f67-876e-7000d5833d82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb6523bb-dd4qf\" (UID: \"6fb6d58d-39c4-4f67-876e-7000d5833d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb6523bb-dd4qf" Nov 03 21:08:58.558297 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:58.558230 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcxv7\" (UniqueName: \"kubernetes.io/projected/99b7150f-e466-4734-97db-cde9002e38b0-kube-api-access-zcxv7\") pod \"maas-api-revoke-sub-346e532f-lztv8\" (UID: \"99b7150f-e466-4734-97db-cde9002e38b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-346e532f-lztv8" Nov 03 21:08:58.558297 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:58.558283 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fb6d58d-39c4-4f67-876e-7000d5833d82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb6523bb-dd4qf\" (UID: \"6fb6d58d-39c4-4f67-876e-7000d5833d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb6523bb-dd4qf" Nov 03 21:08:58.558486 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:58.558465 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjqc5\" (UniqueName: \"kubernetes.io/projected/6fb6d58d-39c4-4f67-876e-7000d5833d82-kube-api-access-fjqc5\") pod \"maas-api-revoke-sub-eb6523bb-dd4qf\" (UID: \"6fb6d58d-39c4-4f67-876e-7000d5833d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb6523bb-dd4qf" Nov 03 21:08:58.558569 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:58.558520 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99b7150f-e466-4734-97db-cde9002e38b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-346e532f-lztv8\" (UID: \"99b7150f-e466-4734-97db-cde9002e38b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-346e532f-lztv8" Nov 03 21:08:58.558628 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:58.558603 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99b7150f-e466-4734-97db-cde9002e38b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-346e532f-lztv8\" (UID: \"99b7150f-e466-4734-97db-cde9002e38b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-346e532f-lztv8" Nov 03 21:08:58.567359 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:58.567331 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-346e532f-lztv8"] Nov 03 21:08:58.660242 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:58.660206 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fb6d58d-39c4-4f67-876e-7000d5833d82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb6523bb-dd4qf\" (UID: \"6fb6d58d-39c4-4f67-876e-7000d5833d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb6523bb-dd4qf" Nov 03 21:08:58.660480 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:58.660326 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjqc5\" (UniqueName: \"kubernetes.io/projected/6fb6d58d-39c4-4f67-876e-7000d5833d82-kube-api-access-fjqc5\") pod \"maas-api-revoke-sub-eb6523bb-dd4qf\" (UID: \"6fb6d58d-39c4-4f67-876e-7000d5833d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb6523bb-dd4qf" Nov 03 21:08:58.660480 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:58.660367 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99b7150f-e466-4734-97db-cde9002e38b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-346e532f-lztv8\" (UID: \"99b7150f-e466-4734-97db-cde9002e38b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-346e532f-lztv8" Nov 03 21:08:58.660480 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:58.660448 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99b7150f-e466-4734-97db-cde9002e38b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-346e532f-lztv8\" (UID: \"99b7150f-e466-4734-97db-cde9002e38b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-346e532f-lztv8" Nov 03 21:08:58.660659 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:58.660573 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fb6d58d-39c4-4f67-876e-7000d5833d82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb6523bb-dd4qf\" (UID: \"6fb6d58d-39c4-4f67-876e-7000d5833d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb6523bb-dd4qf" Nov 03 21:08:58.660711 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:58.660685 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcxv7\" (UniqueName: \"kubernetes.io/projected/99b7150f-e466-4734-97db-cde9002e38b0-kube-api-access-zcxv7\") pod \"maas-api-revoke-sub-346e532f-lztv8\" (UID: \"99b7150f-e466-4734-97db-cde9002e38b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-346e532f-lztv8" Nov 03 21:08:58.661118 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:58.661087 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99b7150f-e466-4734-97db-cde9002e38b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-346e532f-lztv8\" (UID: \"99b7150f-e466-4734-97db-cde9002e38b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-346e532f-lztv8" Nov 03 21:08:58.661240 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:58.661088 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fb6d58d-39c4-4f67-876e-7000d5833d82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb6523bb-dd4qf\" (UID: \"6fb6d58d-39c4-4f67-876e-7000d5833d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb6523bb-dd4qf" Nov 03 21:08:58.663130 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:58.663101 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fb6d58d-39c4-4f67-876e-7000d5833d82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb6523bb-dd4qf\" (UID: \"6fb6d58d-39c4-4f67-876e-7000d5833d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb6523bb-dd4qf" Nov 03 21:08:58.663246 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:58.663128 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99b7150f-e466-4734-97db-cde9002e38b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-346e532f-lztv8\" (UID: \"99b7150f-e466-4734-97db-cde9002e38b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-346e532f-lztv8" Nov 03 21:08:58.739986 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:58.739911 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcxv7\" (UniqueName: \"kubernetes.io/projected/99b7150f-e466-4734-97db-cde9002e38b0-kube-api-access-zcxv7\") pod \"maas-api-revoke-sub-346e532f-lztv8\" (UID: \"99b7150f-e466-4734-97db-cde9002e38b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-346e532f-lztv8" Nov 03 21:08:58.740126 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:58.740074 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjqc5\" (UniqueName: \"kubernetes.io/projected/6fb6d58d-39c4-4f67-876e-7000d5833d82-kube-api-access-fjqc5\") pod \"maas-api-revoke-sub-eb6523bb-dd4qf\" (UID: \"6fb6d58d-39c4-4f67-876e-7000d5833d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb6523bb-dd4qf" Nov 03 21:08:58.758125 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:58.758093 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb6523bb-dd4qf" Nov 03 21:08:58.822784 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:58.822742 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-346e532f-lztv8" Nov 03 21:08:59.146976 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:08:59.146946 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fb6d58d_39c4_4f67_876e_7000d5833d82.slice/crio-c3dd604355a771a30ae42e2c3d68f64a6fae3d359f49542a72b47163da3bbe02 WatchSource:0}: Error finding container c3dd604355a771a30ae42e2c3d68f64a6fae3d359f49542a72b47163da3bbe02: Status 404 returned error can't find the container with id c3dd604355a771a30ae42e2c3d68f64a6fae3d359f49542a72b47163da3bbe02 Nov 03 21:08:59.147476 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:59.147445 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb6523bb-dd4qf"] Nov 03 21:08:59.256456 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:59.253343 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-346e532f-lztv8"] Nov 03 21:08:59.256456 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:08:59.253452 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99b7150f_e466_4734_97db_cde9002e38b0.slice/crio-c4bd7a64d5b291a7bb43a010eedccd27f4adc3b8c7489aa98736bc561de21593 WatchSource:0}: Error finding container c4bd7a64d5b291a7bb43a010eedccd27f4adc3b8c7489aa98736bc561de21593: Status 404 returned error can't find the container with id c4bd7a64d5b291a7bb43a010eedccd27f4adc3b8c7489aa98736bc561de21593 Nov 03 21:08:59.385778 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:59.385735 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-346e532f-lztv8" event={"ID":"99b7150f-e466-4734-97db-cde9002e38b0","Type":"ContainerStarted","Data":"c4bd7a64d5b291a7bb43a010eedccd27f4adc3b8c7489aa98736bc561de21593"} Nov 03 21:08:59.387476 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:59.387397 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb6523bb-dd4qf" event={"ID":"6fb6d58d-39c4-4f67-876e-7000d5833d82","Type":"ContainerStarted","Data":"c3dd604355a771a30ae42e2c3d68f64a6fae3d359f49542a72b47163da3bbe02"} Nov 03 21:08:59.994606 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:08:59.994266 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x" Nov 03 21:09:00.105679 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.105637 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1920ffc0-36e7-4d99-9410-444ad24afd9c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1920ffc0-36e7-4d99-9410-444ad24afd9c-workload-socket\") pod \"1920ffc0-36e7-4d99-9410-444ad24afd9c\" (UID: \"1920ffc0-36e7-4d99-9410-444ad24afd9c\") " Nov 03 21:09:00.105884 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.105727 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1920ffc0-36e7-4d99-9410-444ad24afd9c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1920ffc0-36e7-4d99-9410-444ad24afd9c-istiod-ca-cert\") pod \"1920ffc0-36e7-4d99-9410-444ad24afd9c\" (UID: \"1920ffc0-36e7-4d99-9410-444ad24afd9c\") " Nov 03 21:09:00.105884 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.105754 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1920ffc0-36e7-4d99-9410-444ad24afd9c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1920ffc0-36e7-4d99-9410-444ad24afd9c-istio-data\") pod \"1920ffc0-36e7-4d99-9410-444ad24afd9c\" (UID: \"1920ffc0-36e7-4d99-9410-444ad24afd9c\") " Nov 03 21:09:00.105884 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.105861 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1920ffc0-36e7-4d99-9410-444ad24afd9c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1920ffc0-36e7-4d99-9410-444ad24afd9c-istio-podinfo\") pod \"1920ffc0-36e7-4d99-9410-444ad24afd9c\" (UID: \"1920ffc0-36e7-4d99-9410-444ad24afd9c\") " Nov 03 21:09:00.106063 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.105948 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1920ffc0-36e7-4d99-9410-444ad24afd9c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1920ffc0-36e7-4d99-9410-444ad24afd9c-credential-socket\") pod \"1920ffc0-36e7-4d99-9410-444ad24afd9c\" (UID: \"1920ffc0-36e7-4d99-9410-444ad24afd9c\") " Nov 03 21:09:00.106063 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.105985 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1920ffc0-36e7-4d99-9410-444ad24afd9c-workload-socket" pod "1920ffc0-36e7-4d99-9410-444ad24afd9c" (UID: "1920ffc0-36e7-4d99-9410-444ad24afd9c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:00.106063 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.106058 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1920ffc0-36e7-4d99-9410-444ad24afd9c-istio-token\" (UniqueName: \"kubernetes.io/projected/1920ffc0-36e7-4d99-9410-444ad24afd9c-istio-token\") pod \"1920ffc0-36e7-4d99-9410-444ad24afd9c\" (UID: \"1920ffc0-36e7-4d99-9410-444ad24afd9c\") " Nov 03 21:09:00.106230 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.106093 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1920ffc0-36e7-4d99-9410-444ad24afd9c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1920ffc0-36e7-4d99-9410-444ad24afd9c-istio-envoy\") pod \"1920ffc0-36e7-4d99-9410-444ad24afd9c\" (UID: \"1920ffc0-36e7-4d99-9410-444ad24afd9c\") " Nov 03 21:09:00.106230 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.106122 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1920ffc0-36e7-4d99-9410-444ad24afd9c-kube-api-access-gj55h\" (UniqueName: \"kubernetes.io/projected/1920ffc0-36e7-4d99-9410-444ad24afd9c-kube-api-access-gj55h\") pod \"1920ffc0-36e7-4d99-9410-444ad24afd9c\" (UID: \"1920ffc0-36e7-4d99-9410-444ad24afd9c\") " Nov 03 21:09:00.106230 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.106198 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1920ffc0-36e7-4d99-9410-444ad24afd9c-istio-data" pod "1920ffc0-36e7-4d99-9410-444ad24afd9c" (UID: "1920ffc0-36e7-4d99-9410-444ad24afd9c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:00.106230 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.106219 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1920ffc0-36e7-4d99-9410-444ad24afd9c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1920ffc0-36e7-4d99-9410-444ad24afd9c-workload-certs\") pod \"1920ffc0-36e7-4d99-9410-444ad24afd9c\" (UID: \"1920ffc0-36e7-4d99-9410-444ad24afd9c\") " Nov 03 21:09:00.106592 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.106569 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1920ffc0-36e7-4d99-9410-444ad24afd9c-istiod-ca-cert" pod "1920ffc0-36e7-4d99-9410-444ad24afd9c" (UID: "1920ffc0-36e7-4d99-9410-444ad24afd9c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:09:00.107131 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.107070 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1920ffc0-36e7-4d99-9410-444ad24afd9c-workload-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:00.107131 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.107093 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1920ffc0-36e7-4d99-9410-444ad24afd9c-istiod-ca-cert\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:00.107131 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.107106 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1920ffc0-36e7-4d99-9410-444ad24afd9c-istio-data\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:00.107343 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.107179 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1920ffc0-36e7-4d99-9410-444ad24afd9c-credential-socket" pod "1920ffc0-36e7-4d99-9410-444ad24afd9c" (UID: "1920ffc0-36e7-4d99-9410-444ad24afd9c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:00.107502 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.107480 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1920ffc0-36e7-4d99-9410-444ad24afd9c-workload-certs" pod "1920ffc0-36e7-4d99-9410-444ad24afd9c" (UID: "1920ffc0-36e7-4d99-9410-444ad24afd9c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:00.112663 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.112627 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1920ffc0-36e7-4d99-9410-444ad24afd9c-istio-envoy" pod "1920ffc0-36e7-4d99-9410-444ad24afd9c" (UID: "1920ffc0-36e7-4d99-9410-444ad24afd9c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:00.114987 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.114953 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1920ffc0-36e7-4d99-9410-444ad24afd9c-istio-token" pod "1920ffc0-36e7-4d99-9410-444ad24afd9c" (UID: "1920ffc0-36e7-4d99-9410-444ad24afd9c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:09:00.115503 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.115476 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1920ffc0-36e7-4d99-9410-444ad24afd9c-istio-podinfo" pod "1920ffc0-36e7-4d99-9410-444ad24afd9c" (UID: "1920ffc0-36e7-4d99-9410-444ad24afd9c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:09:00.115877 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.115842 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1920ffc0-36e7-4d99-9410-444ad24afd9c-kube-api-access-gj55h" pod "1920ffc0-36e7-4d99-9410-444ad24afd9c" (UID: "1920ffc0-36e7-4d99-9410-444ad24afd9c"). InnerVolumeSpecName "kube-api-access-gj55h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:09:00.169825 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.169798 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d" Nov 03 21:09:00.209709 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.209662 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1920ffc0-36e7-4d99-9410-444ad24afd9c-istio-podinfo\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:00.209709 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.209707 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1920ffc0-36e7-4d99-9410-444ad24afd9c-credential-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:00.209968 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.209725 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1920ffc0-36e7-4d99-9410-444ad24afd9c-istio-token\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:00.209968 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.209739 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1920ffc0-36e7-4d99-9410-444ad24afd9c-istio-envoy\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:00.209968 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.209753 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj55h\" (UniqueName: \"kubernetes.io/projected/1920ffc0-36e7-4d99-9410-444ad24afd9c-kube-api-access-gj55h\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:00.209968 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.209770 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1920ffc0-36e7-4d99-9410-444ad24afd9c-workload-certs\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:00.228290 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.228258 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh" Nov 03 21:09:00.285712 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.285678 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd" Nov 03 21:09:00.301924 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.301832 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28" Nov 03 21:09:00.308947 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.308865 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx" Nov 03 21:09:00.312177 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.310561 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1-workload-certs\") pod \"17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1\" (UID: \"17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1\") " Nov 03 21:09:00.312177 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.310605 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9103e89-52f2-4022-bd9b-dc554c82f479-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9103e89-52f2-4022-bd9b-dc554c82f479-credential-socket\") pod \"d9103e89-52f2-4022-bd9b-dc554c82f479\" (UID: \"d9103e89-52f2-4022-bd9b-dc554c82f479\") " Nov 03 21:09:00.312177 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.310640 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9103e89-52f2-4022-bd9b-dc554c82f479-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9103e89-52f2-4022-bd9b-dc554c82f479-istio-envoy\") pod \"d9103e89-52f2-4022-bd9b-dc554c82f479\" (UID: \"d9103e89-52f2-4022-bd9b-dc554c82f479\") " Nov 03 21:09:00.312177 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.310684 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1-workload-socket\") pod \"17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1\" (UID: \"17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1\") " Nov 03 21:09:00.312177 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.310706 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9103e89-52f2-4022-bd9b-dc554c82f479-istio-token\" (UniqueName: \"kubernetes.io/projected/d9103e89-52f2-4022-bd9b-dc554c82f479-istio-token\") pod \"d9103e89-52f2-4022-bd9b-dc554c82f479\" (UID: \"d9103e89-52f2-4022-bd9b-dc554c82f479\") " Nov 03 21:09:00.312177 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.310747 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9103e89-52f2-4022-bd9b-dc554c82f479-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9103e89-52f2-4022-bd9b-dc554c82f479-istio-data\") pod \"d9103e89-52f2-4022-bd9b-dc554c82f479\" (UID: \"d9103e89-52f2-4022-bd9b-dc554c82f479\") " Nov 03 21:09:00.312177 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.310777 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1-istio-data\") pod \"17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1\" (UID: \"17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1\") " Nov 03 21:09:00.312177 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.310809 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1-istio-envoy\") pod \"17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1\" (UID: \"17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1\") " Nov 03 21:09:00.312177 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.310841 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1-istio-podinfo\") pod \"17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1\" (UID: \"17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1\") " Nov 03 21:09:00.312177 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.310895 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1-kube-api-access-mlqfk\" (UniqueName: \"kubernetes.io/projected/17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1-kube-api-access-mlqfk\") pod \"17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1\" (UID: \"17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1\") " Nov 03 21:09:00.312177 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.310941 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1-credential-socket\") pod \"17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1\" (UID: \"17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1\") " Nov 03 21:09:00.312177 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.310992 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9103e89-52f2-4022-bd9b-dc554c82f479-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9103e89-52f2-4022-bd9b-dc554c82f479-istiod-ca-cert\") pod \"d9103e89-52f2-4022-bd9b-dc554c82f479\" (UID: \"d9103e89-52f2-4022-bd9b-dc554c82f479\") " Nov 03 21:09:00.312177 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.311056 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9103e89-52f2-4022-bd9b-dc554c82f479-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9103e89-52f2-4022-bd9b-dc554c82f479-workload-socket\") pod \"d9103e89-52f2-4022-bd9b-dc554c82f479\" (UID: \"d9103e89-52f2-4022-bd9b-dc554c82f479\") " Nov 03 21:09:00.312177 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.311076 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9103e89-52f2-4022-bd9b-dc554c82f479-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9103e89-52f2-4022-bd9b-dc554c82f479-workload-certs\") pod \"d9103e89-52f2-4022-bd9b-dc554c82f479\" (UID: \"d9103e89-52f2-4022-bd9b-dc554c82f479\") " Nov 03 21:09:00.312177 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.311118 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9103e89-52f2-4022-bd9b-dc554c82f479-kube-api-access-lqzwj\" (UniqueName: \"kubernetes.io/projected/d9103e89-52f2-4022-bd9b-dc554c82f479-kube-api-access-lqzwj\") pod \"d9103e89-52f2-4022-bd9b-dc554c82f479\" (UID: \"d9103e89-52f2-4022-bd9b-dc554c82f479\") " Nov 03 21:09:00.312177 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.311154 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1-istio-token\" (UniqueName: \"kubernetes.io/projected/17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1-istio-token\") pod \"17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1\" (UID: \"17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1\") " Nov 03 21:09:00.312177 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.311170 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d9103e89-52f2-4022-bd9b-dc554c82f479-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9103e89-52f2-4022-bd9b-dc554c82f479-istio-podinfo\") pod \"d9103e89-52f2-4022-bd9b-dc554c82f479\" (UID: \"d9103e89-52f2-4022-bd9b-dc554c82f479\") " Nov 03 21:09:00.312177 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.311213 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1-istiod-ca-cert\") pod \"17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1\" (UID: \"17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1\") " Nov 03 21:09:00.315080 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.314818 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1-istiod-ca-cert" pod "17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1" (UID: "17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:09:00.318215 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.316200 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9103e89-52f2-4022-bd9b-dc554c82f479-workload-certs" pod "d9103e89-52f2-4022-bd9b-dc554c82f479" (UID: "d9103e89-52f2-4022-bd9b-dc554c82f479"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:00.318215 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.316490 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1-credential-socket" pod "17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1" (UID: "17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:00.318215 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.316775 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9103e89-52f2-4022-bd9b-dc554c82f479-istiod-ca-cert" pod "d9103e89-52f2-4022-bd9b-dc554c82f479" (UID: "d9103e89-52f2-4022-bd9b-dc554c82f479"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:09:00.318215 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.317014 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9103e89-52f2-4022-bd9b-dc554c82f479-workload-socket" pod "d9103e89-52f2-4022-bd9b-dc554c82f479" (UID: "d9103e89-52f2-4022-bd9b-dc554c82f479"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:00.320166 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.318625 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9103e89-52f2-4022-bd9b-dc554c82f479-istio-token" pod "d9103e89-52f2-4022-bd9b-dc554c82f479" (UID: "d9103e89-52f2-4022-bd9b-dc554c82f479"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:09:00.320166 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.319267 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1-istio-token" pod "17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1" (UID: "17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:09:00.320627 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.320351 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1-istio-podinfo" pod "17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1" (UID: "17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:09:00.320912 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.320757 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9103e89-52f2-4022-bd9b-dc554c82f479-credential-socket" pod "d9103e89-52f2-4022-bd9b-dc554c82f479" (UID: "d9103e89-52f2-4022-bd9b-dc554c82f479"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:00.321286 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.321026 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1-workload-certs" pod "17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1" (UID: "17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:00.321664 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.321595 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9103e89-52f2-4022-bd9b-dc554c82f479-kube-api-access-lqzwj" pod "d9103e89-52f2-4022-bd9b-dc554c82f479" (UID: "d9103e89-52f2-4022-bd9b-dc554c82f479"). InnerVolumeSpecName "kube-api-access-lqzwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:09:00.321664 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.321609 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1-istio-data" pod "17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1" (UID: "17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:00.322063 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.322031 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9103e89-52f2-4022-bd9b-dc554c82f479-istio-data" pod "d9103e89-52f2-4022-bd9b-dc554c82f479" (UID: "d9103e89-52f2-4022-bd9b-dc554c82f479"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:00.322360 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.322343 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1-workload-socket" pod "17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1" (UID: "17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:00.323890 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.323768 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d9103e89-52f2-4022-bd9b-dc554c82f479-istio-podinfo" pod "d9103e89-52f2-4022-bd9b-dc554c82f479" (UID: "d9103e89-52f2-4022-bd9b-dc554c82f479"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:09:00.323890 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.323860 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1-kube-api-access-mlqfk" pod "17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1" (UID: "17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1"). InnerVolumeSpecName "kube-api-access-mlqfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:09:00.324066 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.323972 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1-istio-envoy" pod "17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1" (UID: "17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:00.325493 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.325412 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9103e89-52f2-4022-bd9b-dc554c82f479-istio-envoy" pod "d9103e89-52f2-4022-bd9b-dc554c82f479" (UID: "d9103e89-52f2-4022-bd9b-dc554c82f479"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:00.340119 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.340035 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd" Nov 03 21:09:00.401130 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.401088 4305 generic.go:356] "Generic (PLEG): container finished" podID="58829723-0628-457e-80b8-f81d15ecfe2e" containerID="bf7eed539704b28314d8666f8366e928a9a1acdf0bd6773de39b3c6c370f01a9" exitCode=0 Nov 03 21:09:00.401328 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.401156 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28" Nov 03 21:09:00.401328 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.401187 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28" event={"ID":"58829723-0628-457e-80b8-f81d15ecfe2e","Type":"ContainerDied","Data":"bf7eed539704b28314d8666f8366e928a9a1acdf0bd6773de39b3c6c370f01a9"} Nov 03 21:09:00.401328 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.401264 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28" event={"ID":"58829723-0628-457e-80b8-f81d15ecfe2e","Type":"ContainerDied","Data":"a5e133206cfceedb5ba0470db9c881852cfb9e7ee52dee7732c7be09df305869"} Nov 03 21:09:00.401328 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.401287 4305 scope.go:122] "RemoveContainer" containerID="bf7eed539704b28314d8666f8366e928a9a1acdf0bd6773de39b3c6c370f01a9" Nov 03 21:09:00.402836 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.402808 4305 generic.go:356] "Generic (PLEG): container finished" podID="1920ffc0-36e7-4d99-9410-444ad24afd9c" containerID="dfbf78fe82e05e2a90b38ad21dac6ac7e2793aa394447992c7651e802a95478d" exitCode=0 Nov 03 21:09:00.402984 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.402838 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x" event={"ID":"1920ffc0-36e7-4d99-9410-444ad24afd9c","Type":"ContainerDied","Data":"dfbf78fe82e05e2a90b38ad21dac6ac7e2793aa394447992c7651e802a95478d"} Nov 03 21:09:00.402984 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.402875 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x" Nov 03 21:09:00.402984 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.402881 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x" event={"ID":"1920ffc0-36e7-4d99-9410-444ad24afd9c","Type":"ContainerDied","Data":"e27c9be914b46c36f33b2a1a0878b4957735860465de2ef23f15725c354f363d"} Nov 03 21:09:00.404531 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.404504 4305 generic.go:356] "Generic (PLEG): container finished" podID="17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1" containerID="ea7e5f654bccbde6382c9f018f3f2ec5f2bc78514d2af0f94c66559e0d9aae46" exitCode=0 Nov 03 21:09:00.407338 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.404626 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh" Nov 03 21:09:00.407338 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.404630 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh" event={"ID":"17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1","Type":"ContainerDied","Data":"ea7e5f654bccbde6382c9f018f3f2ec5f2bc78514d2af0f94c66559e0d9aae46"} Nov 03 21:09:00.407338 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.404795 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh" event={"ID":"17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1","Type":"ContainerDied","Data":"4b7db1844b8f7e2c6850a6de34987e8a0be37f36e4ab613887063b938b0024de"} Nov 03 21:09:00.407338 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.406671 4305 generic.go:356] "Generic (PLEG): container finished" podID="6fb6d58d-39c4-4f67-876e-7000d5833d82" containerID="0a8c351b3f7b2fd30e038ca961d55d967299ae906ecc77687ba09a2d3804c15d" exitCode=0 Nov 03 21:09:00.407338 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.406759 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb6523bb-dd4qf" event={"ID":"6fb6d58d-39c4-4f67-876e-7000d5833d82","Type":"ContainerDied","Data":"0a8c351b3f7b2fd30e038ca961d55d967299ae906ecc77687ba09a2d3804c15d"} Nov 03 21:09:00.409016 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.408985 4305 generic.go:356] "Generic (PLEG): container finished" podID="36f2d843-7d6a-4606-b6b3-6904a2b6cc29" containerID="8818f76d7f9e22d97b1005dad732e19ae4011a562bc75443701eec18a1f9726e" exitCode=0 Nov 03 21:09:00.409137 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.409078 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx" event={"ID":"36f2d843-7d6a-4606-b6b3-6904a2b6cc29","Type":"ContainerDied","Data":"8818f76d7f9e22d97b1005dad732e19ae4011a562bc75443701eec18a1f9726e"} Nov 03 21:09:00.409137 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.409119 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx" event={"ID":"36f2d843-7d6a-4606-b6b3-6904a2b6cc29","Type":"ContainerDied","Data":"67144fb97d0431f7c086651fcdb49927a0bae010aebbd211c9419b7c124021fc"} Nov 03 21:09:00.409263 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.409209 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx" Nov 03 21:09:00.410703 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.410676 4305 generic.go:356] "Generic (PLEG): container finished" podID="0ff154f8-a2c6-49a4-bbc8-f5026ec757bd" containerID="4d111a68c0e46f09480dfdfe969cb1010022c19b4dd169cff64e9ee661f2334c" exitCode=0 Nov 03 21:09:00.410820 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.410757 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd" event={"ID":"0ff154f8-a2c6-49a4-bbc8-f5026ec757bd","Type":"ContainerDied","Data":"4d111a68c0e46f09480dfdfe969cb1010022c19b4dd169cff64e9ee661f2334c"} Nov 03 21:09:00.410820 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.410798 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd" event={"ID":"0ff154f8-a2c6-49a4-bbc8-f5026ec757bd","Type":"ContainerDied","Data":"57c630e392d5609a81722ba5ba748f66d488fcc6996229f9b33f31bceef7eac2"} Nov 03 21:09:00.410947 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.410888 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd" Nov 03 21:09:00.412576 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.412550 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-346e532f-lztv8" event={"ID":"99b7150f-e466-4734-97db-cde9002e38b0","Type":"ContainerStarted","Data":"4cfb4c063c1bbbe62e1ca1fc0192559c0707b52c482cb306758bd0397f8f5359"} Nov 03 21:09:00.414828 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.414804 4305 scope.go:122] "RemoveContainer" containerID="10d10b551cde5c66d72ab000240f018c97e9ffa11aafc083cfab2f0a0f710fe3" Nov 03 21:09:00.414923 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.414845 4305 generic.go:356] "Generic (PLEG): container finished" podID="d9103e89-52f2-4022-bd9b-dc554c82f479" containerID="d950b9777f3fe72d899dfe98f108d37bb059466e0e8b4c1b5216870ecd4bafcc" exitCode=0 Nov 03 21:09:00.414923 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.414912 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d" Nov 03 21:09:00.415014 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.414949 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d" event={"ID":"d9103e89-52f2-4022-bd9b-dc554c82f479","Type":"ContainerDied","Data":"d950b9777f3fe72d899dfe98f108d37bb059466e0e8b4c1b5216870ecd4bafcc"} Nov 03 21:09:00.415014 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.414989 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d" event={"ID":"d9103e89-52f2-4022-bd9b-dc554c82f479","Type":"ContainerDied","Data":"8090be0882cf15822b6704f55a8398f02bcdea74e215a356c3361cfdc39e69f7"} Nov 03 21:09:00.415086 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.415027 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ff154f8-a2c6-49a4-bbc8-f5026ec757bd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ff154f8-a2c6-49a4-bbc8-f5026ec757bd-istio-envoy\") pod \"0ff154f8-a2c6-49a4-bbc8-f5026ec757bd\" (UID: \"0ff154f8-a2c6-49a4-bbc8-f5026ec757bd\") " Nov 03 21:09:00.415200 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.415122 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ff154f8-a2c6-49a4-bbc8-f5026ec757bd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ff154f8-a2c6-49a4-bbc8-f5026ec757bd-workload-certs\") pod \"0ff154f8-a2c6-49a4-bbc8-f5026ec757bd\" (UID: \"0ff154f8-a2c6-49a4-bbc8-f5026ec757bd\") " Nov 03 21:09:00.415200 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.415161 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58829723-0628-457e-80b8-f81d15ecfe2e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58829723-0628-457e-80b8-f81d15ecfe2e-credential-socket\") pod \"58829723-0628-457e-80b8-f81d15ecfe2e\" (UID: \"58829723-0628-457e-80b8-f81d15ecfe2e\") " Nov 03 21:09:00.415200 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.415190 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36f2d843-7d6a-4606-b6b3-6904a2b6cc29-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36f2d843-7d6a-4606-b6b3-6904a2b6cc29-workload-socket\") pod \"36f2d843-7d6a-4606-b6b3-6904a2b6cc29\" (UID: \"36f2d843-7d6a-4606-b6b3-6904a2b6cc29\") " Nov 03 21:09:00.415362 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.415233 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ff154f8-a2c6-49a4-bbc8-f5026ec757bd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ff154f8-a2c6-49a4-bbc8-f5026ec757bd-istiod-ca-cert\") pod \"0ff154f8-a2c6-49a4-bbc8-f5026ec757bd\" (UID: \"0ff154f8-a2c6-49a4-bbc8-f5026ec757bd\") " Nov 03 21:09:00.415362 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.415264 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ff154f8-a2c6-49a4-bbc8-f5026ec757bd-istio-token\" (UniqueName: \"kubernetes.io/projected/0ff154f8-a2c6-49a4-bbc8-f5026ec757bd-istio-token\") pod \"0ff154f8-a2c6-49a4-bbc8-f5026ec757bd\" (UID: \"0ff154f8-a2c6-49a4-bbc8-f5026ec757bd\") " Nov 03 21:09:00.415362 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.415328 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36f2d843-7d6a-4606-b6b3-6904a2b6cc29-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36f2d843-7d6a-4606-b6b3-6904a2b6cc29-credential-socket\") pod \"36f2d843-7d6a-4606-b6b3-6904a2b6cc29\" (UID: \"36f2d843-7d6a-4606-b6b3-6904a2b6cc29\") " Nov 03 21:09:00.415968 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.415372 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/36f2d843-7d6a-4606-b6b3-6904a2b6cc29-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36f2d843-7d6a-4606-b6b3-6904a2b6cc29-istio-podinfo\") pod \"36f2d843-7d6a-4606-b6b3-6904a2b6cc29\" (UID: \"36f2d843-7d6a-4606-b6b3-6904a2b6cc29\") " Nov 03 21:09:00.415968 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.415403 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58829723-0628-457e-80b8-f81d15ecfe2e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58829723-0628-457e-80b8-f81d15ecfe2e-istiod-ca-cert\") pod \"58829723-0628-457e-80b8-f81d15ecfe2e\" (UID: \"58829723-0628-457e-80b8-f81d15ecfe2e\") " Nov 03 21:09:00.415968 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.415505 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ff154f8-a2c6-49a4-bbc8-f5026ec757bd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ff154f8-a2c6-49a4-bbc8-f5026ec757bd-istio-data\") pod \"0ff154f8-a2c6-49a4-bbc8-f5026ec757bd\" (UID: \"0ff154f8-a2c6-49a4-bbc8-f5026ec757bd\") " Nov 03 21:09:00.415968 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.415504 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58829723-0628-457e-80b8-f81d15ecfe2e-credential-socket" pod "58829723-0628-457e-80b8-f81d15ecfe2e" (UID: "58829723-0628-457e-80b8-f81d15ecfe2e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:00.415968 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.415540 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36f2d843-7d6a-4606-b6b3-6904a2b6cc29-workload-socket" pod "36f2d843-7d6a-4606-b6b3-6904a2b6cc29" (UID: "36f2d843-7d6a-4606-b6b3-6904a2b6cc29"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:00.415968 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.415573 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36f2d843-7d6a-4606-b6b3-6904a2b6cc29-kube-api-access-2tkw7\" (UniqueName: \"kubernetes.io/projected/36f2d843-7d6a-4606-b6b3-6904a2b6cc29-kube-api-access-2tkw7\") pod \"36f2d843-7d6a-4606-b6b3-6904a2b6cc29\" (UID: \"36f2d843-7d6a-4606-b6b3-6904a2b6cc29\") " Nov 03 21:09:00.415968 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.415607 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58829723-0628-457e-80b8-f81d15ecfe2e-istio-token\" (UniqueName: \"kubernetes.io/projected/58829723-0628-457e-80b8-f81d15ecfe2e-istio-token\") pod \"58829723-0628-457e-80b8-f81d15ecfe2e\" (UID: \"58829723-0628-457e-80b8-f81d15ecfe2e\") " Nov 03 21:09:00.415968 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.415678 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58829723-0628-457e-80b8-f81d15ecfe2e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58829723-0628-457e-80b8-f81d15ecfe2e-workload-socket\") pod \"58829723-0628-457e-80b8-f81d15ecfe2e\" (UID: \"58829723-0628-457e-80b8-f81d15ecfe2e\") " Nov 03 21:09:00.415968 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.415781 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36f2d843-7d6a-4606-b6b3-6904a2b6cc29-istio-token\" (UniqueName: \"kubernetes.io/projected/36f2d843-7d6a-4606-b6b3-6904a2b6cc29-istio-token\") pod \"36f2d843-7d6a-4606-b6b3-6904a2b6cc29\" (UID: \"36f2d843-7d6a-4606-b6b3-6904a2b6cc29\") " Nov 03 21:09:00.415968 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.415839 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36f2d843-7d6a-4606-b6b3-6904a2b6cc29-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36f2d843-7d6a-4606-b6b3-6904a2b6cc29-workload-certs\") pod \"36f2d843-7d6a-4606-b6b3-6904a2b6cc29\" (UID: \"36f2d843-7d6a-4606-b6b3-6904a2b6cc29\") " Nov 03 21:09:00.415968 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.415878 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58829723-0628-457e-80b8-f81d15ecfe2e-kube-api-access-l9bmc\" (UniqueName: \"kubernetes.io/projected/58829723-0628-457e-80b8-f81d15ecfe2e-kube-api-access-l9bmc\") pod \"58829723-0628-457e-80b8-f81d15ecfe2e\" (UID: \"58829723-0628-457e-80b8-f81d15ecfe2e\") " Nov 03 21:09:00.415968 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.415932 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36f2d843-7d6a-4606-b6b3-6904a2b6cc29-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36f2d843-7d6a-4606-b6b3-6904a2b6cc29-istio-envoy\") pod \"36f2d843-7d6a-4606-b6b3-6904a2b6cc29\" (UID: \"36f2d843-7d6a-4606-b6b3-6904a2b6cc29\") " Nov 03 21:09:00.416581 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.415992 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58829723-0628-457e-80b8-f81d15ecfe2e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58829723-0628-457e-80b8-f81d15ecfe2e-istio-data\") pod \"58829723-0628-457e-80b8-f81d15ecfe2e\" (UID: \"58829723-0628-457e-80b8-f81d15ecfe2e\") " Nov 03 21:09:00.463742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.422322 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ff154f8-a2c6-49a4-bbc8-f5026ec757bd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff154f8-a2c6-49a4-bbc8-f5026ec757bd-credential-socket\") pod \"0ff154f8-a2c6-49a4-bbc8-f5026ec757bd\" (UID: \"0ff154f8-a2c6-49a4-bbc8-f5026ec757bd\") " Nov 03 21:09:00.463742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.422408 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/58829723-0628-457e-80b8-f81d15ecfe2e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58829723-0628-457e-80b8-f81d15ecfe2e-istio-podinfo\") pod \"58829723-0628-457e-80b8-f81d15ecfe2e\" (UID: \"58829723-0628-457e-80b8-f81d15ecfe2e\") " Nov 03 21:09:00.463742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.422466 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58829723-0628-457e-80b8-f81d15ecfe2e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58829723-0628-457e-80b8-f81d15ecfe2e-istio-envoy\") pod \"58829723-0628-457e-80b8-f81d15ecfe2e\" (UID: \"58829723-0628-457e-80b8-f81d15ecfe2e\") " Nov 03 21:09:00.463742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.422498 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36f2d843-7d6a-4606-b6b3-6904a2b6cc29-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36f2d843-7d6a-4606-b6b3-6904a2b6cc29-istio-data\") pod \"36f2d843-7d6a-4606-b6b3-6904a2b6cc29\" (UID: \"36f2d843-7d6a-4606-b6b3-6904a2b6cc29\") " Nov 03 21:09:00.463742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.422529 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36f2d843-7d6a-4606-b6b3-6904a2b6cc29-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36f2d843-7d6a-4606-b6b3-6904a2b6cc29-istiod-ca-cert\") pod \"36f2d843-7d6a-4606-b6b3-6904a2b6cc29\" (UID: \"36f2d843-7d6a-4606-b6b3-6904a2b6cc29\") " Nov 03 21:09:00.463742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.422584 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ff154f8-a2c6-49a4-bbc8-f5026ec757bd-kube-api-access-6tkls\" (UniqueName: \"kubernetes.io/projected/0ff154f8-a2c6-49a4-bbc8-f5026ec757bd-kube-api-access-6tkls\") pod \"0ff154f8-a2c6-49a4-bbc8-f5026ec757bd\" (UID: \"0ff154f8-a2c6-49a4-bbc8-f5026ec757bd\") " Nov 03 21:09:00.463742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.422616 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ff154f8-a2c6-49a4-bbc8-f5026ec757bd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ff154f8-a2c6-49a4-bbc8-f5026ec757bd-istio-podinfo\") pod \"0ff154f8-a2c6-49a4-bbc8-f5026ec757bd\" (UID: \"0ff154f8-a2c6-49a4-bbc8-f5026ec757bd\") " Nov 03 21:09:00.463742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.422704 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58829723-0628-457e-80b8-f81d15ecfe2e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58829723-0628-457e-80b8-f81d15ecfe2e-workload-certs\") pod \"58829723-0628-457e-80b8-f81d15ecfe2e\" (UID: \"58829723-0628-457e-80b8-f81d15ecfe2e\") " Nov 03 21:09:00.463742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.422741 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ff154f8-a2c6-49a4-bbc8-f5026ec757bd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff154f8-a2c6-49a4-bbc8-f5026ec757bd-workload-socket\") pod \"0ff154f8-a2c6-49a4-bbc8-f5026ec757bd\" (UID: \"0ff154f8-a2c6-49a4-bbc8-f5026ec757bd\") " Nov 03 21:09:00.463742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.426796 4305 generic.go:356] "Generic (PLEG): container finished" podID="94bcfeb9-714d-4f25-8ce7-c6c4577ea837" containerID="b1008eb64b520f663b7e82c72eecf11ea5d8037cbd6c6f312cfe85e5a20e97b7" exitCode=0 Nov 03 21:09:00.463742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.426945 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd" event={"ID":"94bcfeb9-714d-4f25-8ce7-c6c4577ea837","Type":"ContainerDied","Data":"b1008eb64b520f663b7e82c72eecf11ea5d8037cbd6c6f312cfe85e5a20e97b7"} Nov 03 21:09:00.463742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.426991 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd" event={"ID":"94bcfeb9-714d-4f25-8ce7-c6c4577ea837","Type":"ContainerDied","Data":"c19d07edeaf6cf3be9b0a38d9ab7456d5e5888983f1a3712393d4063195b3947"} Nov 03 21:09:00.463742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.427180 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd" Nov 03 21:09:00.463742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.427383 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1-istio-podinfo\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:00.463742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.427407 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlqfk\" (UniqueName: \"kubernetes.io/projected/17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1-kube-api-access-mlqfk\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:00.463742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.427424 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1-credential-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:00.463742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.427458 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9103e89-52f2-4022-bd9b-dc554c82f479-istiod-ca-cert\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:00.463742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.427473 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9103e89-52f2-4022-bd9b-dc554c82f479-workload-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:00.463742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.427486 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9103e89-52f2-4022-bd9b-dc554c82f479-workload-certs\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:00.463742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.427499 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58829723-0628-457e-80b8-f81d15ecfe2e-credential-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:00.463742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.427512 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36f2d843-7d6a-4606-b6b3-6904a2b6cc29-workload-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:00.463742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.427525 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqzwj\" (UniqueName: \"kubernetes.io/projected/d9103e89-52f2-4022-bd9b-dc554c82f479-kube-api-access-lqzwj\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:00.463742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.427539 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1-istio-token\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:00.463742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.427552 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9103e89-52f2-4022-bd9b-dc554c82f479-istio-podinfo\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:00.463742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.427568 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1-istiod-ca-cert\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:00.463742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.427581 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1-workload-certs\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:00.463742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.427594 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9103e89-52f2-4022-bd9b-dc554c82f479-credential-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:00.463742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.427607 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9103e89-52f2-4022-bd9b-dc554c82f479-istio-envoy\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:00.463742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.427620 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1-workload-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:00.463742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.427636 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9103e89-52f2-4022-bd9b-dc554c82f479-istio-token\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:00.463742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.427648 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9103e89-52f2-4022-bd9b-dc554c82f479-istio-data\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:00.463742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.427662 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1-istio-data\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:00.463742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.427675 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1-istio-envoy\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:00.463742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.415784 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff154f8-a2c6-49a4-bbc8-f5026ec757bd-istio-data" pod "0ff154f8-a2c6-49a4-bbc8-f5026ec757bd" (UID: "0ff154f8-a2c6-49a4-bbc8-f5026ec757bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:00.463742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.416212 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ff154f8-a2c6-49a4-bbc8-f5026ec757bd-istiod-ca-cert" pod "0ff154f8-a2c6-49a4-bbc8-f5026ec757bd" (UID: "0ff154f8-a2c6-49a4-bbc8-f5026ec757bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:09:00.463742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.427088 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff154f8-a2c6-49a4-bbc8-f5026ec757bd-istio-envoy" pod "0ff154f8-a2c6-49a4-bbc8-f5026ec757bd" (UID: "0ff154f8-a2c6-49a4-bbc8-f5026ec757bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:00.463742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.427785 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58829723-0628-457e-80b8-f81d15ecfe2e-istio-data" pod "58829723-0628-457e-80b8-f81d15ecfe2e" (UID: "58829723-0628-457e-80b8-f81d15ecfe2e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:00.463742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.427867 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff154f8-a2c6-49a4-bbc8-f5026ec757bd-workload-socket" pod "0ff154f8-a2c6-49a4-bbc8-f5026ec757bd" (UID: "0ff154f8-a2c6-49a4-bbc8-f5026ec757bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:00.463742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.428044 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff154f8-a2c6-49a4-bbc8-f5026ec757bd-credential-socket" pod "0ff154f8-a2c6-49a4-bbc8-f5026ec757bd" (UID: "0ff154f8-a2c6-49a4-bbc8-f5026ec757bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:00.463742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.428488 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff154f8-a2c6-49a4-bbc8-f5026ec757bd-workload-certs" pod "0ff154f8-a2c6-49a4-bbc8-f5026ec757bd" (UID: "0ff154f8-a2c6-49a4-bbc8-f5026ec757bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:00.463742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.429720 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36f2d843-7d6a-4606-b6b3-6904a2b6cc29-credential-socket" pod "36f2d843-7d6a-4606-b6b3-6904a2b6cc29" (UID: "36f2d843-7d6a-4606-b6b3-6904a2b6cc29"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:00.463742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.430359 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36f2d843-7d6a-4606-b6b3-6904a2b6cc29-istiod-ca-cert" pod "36f2d843-7d6a-4606-b6b3-6904a2b6cc29" (UID: "36f2d843-7d6a-4606-b6b3-6904a2b6cc29"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:09:00.463742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.438984 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58829723-0628-457e-80b8-f81d15ecfe2e-workload-certs" pod "58829723-0628-457e-80b8-f81d15ecfe2e" (UID: "58829723-0628-457e-80b8-f81d15ecfe2e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:00.463742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.439392 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36f2d843-7d6a-4606-b6b3-6904a2b6cc29-istio-data" pod "36f2d843-7d6a-4606-b6b3-6904a2b6cc29" (UID: "36f2d843-7d6a-4606-b6b3-6904a2b6cc29"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:00.463742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.440503 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58829723-0628-457e-80b8-f81d15ecfe2e-istiod-ca-cert" pod "58829723-0628-457e-80b8-f81d15ecfe2e" (UID: "58829723-0628-457e-80b8-f81d15ecfe2e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:09:00.463742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.442514 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ff154f8-a2c6-49a4-bbc8-f5026ec757bd-istio-token" pod "0ff154f8-a2c6-49a4-bbc8-f5026ec757bd" (UID: "0ff154f8-a2c6-49a4-bbc8-f5026ec757bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:09:00.463742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.443107 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58829723-0628-457e-80b8-f81d15ecfe2e-workload-socket" pod "58829723-0628-457e-80b8-f81d15ecfe2e" (UID: "58829723-0628-457e-80b8-f81d15ecfe2e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:00.463742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.444422 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36f2d843-7d6a-4606-b6b3-6904a2b6cc29-workload-certs" pod "36f2d843-7d6a-4606-b6b3-6904a2b6cc29" (UID: "36f2d843-7d6a-4606-b6b3-6904a2b6cc29"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:00.463742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.446946 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36f2d843-7d6a-4606-b6b3-6904a2b6cc29-kube-api-access-2tkw7" pod "36f2d843-7d6a-4606-b6b3-6904a2b6cc29" (UID: "36f2d843-7d6a-4606-b6b3-6904a2b6cc29"). InnerVolumeSpecName "kube-api-access-2tkw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:09:00.463742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.447377 4305 scope.go:122] "RemoveContainer" containerID="bf7eed539704b28314d8666f8366e928a9a1acdf0bd6773de39b3c6c370f01a9" Nov 03 21:09:00.463742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.447416 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58829723-0628-457e-80b8-f81d15ecfe2e-istio-envoy" pod "58829723-0628-457e-80b8-f81d15ecfe2e" (UID: "58829723-0628-457e-80b8-f81d15ecfe2e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:00.463742 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:09:00.447773 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf7eed539704b28314d8666f8366e928a9a1acdf0bd6773de39b3c6c370f01a9\": container with ID starting with bf7eed539704b28314d8666f8366e928a9a1acdf0bd6773de39b3c6c370f01a9 not found: ID does not exist" containerID="bf7eed539704b28314d8666f8366e928a9a1acdf0bd6773de39b3c6c370f01a9" Nov 03 21:09:00.463742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.447809 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf7eed539704b28314d8666f8366e928a9a1acdf0bd6773de39b3c6c370f01a9"} err="failed to get container status \"bf7eed539704b28314d8666f8366e928a9a1acdf0bd6773de39b3c6c370f01a9\": rpc error: code = NotFound desc = could not find container \"bf7eed539704b28314d8666f8366e928a9a1acdf0bd6773de39b3c6c370f01a9\": container with ID starting with bf7eed539704b28314d8666f8366e928a9a1acdf0bd6773de39b3c6c370f01a9 not found: ID does not exist" Nov 03 21:09:00.463742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.447827 4305 scope.go:122] "RemoveContainer" containerID="10d10b551cde5c66d72ab000240f018c97e9ffa11aafc083cfab2f0a0f710fe3" Nov 03 21:09:00.463742 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:09:00.448152 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10d10b551cde5c66d72ab000240f018c97e9ffa11aafc083cfab2f0a0f710fe3\": container with ID starting with 10d10b551cde5c66d72ab000240f018c97e9ffa11aafc083cfab2f0a0f710fe3 not found: ID does not exist" containerID="10d10b551cde5c66d72ab000240f018c97e9ffa11aafc083cfab2f0a0f710fe3" Nov 03 21:09:00.463742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.448183 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10d10b551cde5c66d72ab000240f018c97e9ffa11aafc083cfab2f0a0f710fe3"} err="failed to get container status \"10d10b551cde5c66d72ab000240f018c97e9ffa11aafc083cfab2f0a0f710fe3\": rpc error: code = NotFound desc = could not find container \"10d10b551cde5c66d72ab000240f018c97e9ffa11aafc083cfab2f0a0f710fe3\": container with ID starting with 10d10b551cde5c66d72ab000240f018c97e9ffa11aafc083cfab2f0a0f710fe3 not found: ID does not exist" Nov 03 21:09:00.463742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.448206 4305 scope.go:122] "RemoveContainer" containerID="dfbf78fe82e05e2a90b38ad21dac6ac7e2793aa394447992c7651e802a95478d" Nov 03 21:09:00.463742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.452842 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/58829723-0628-457e-80b8-f81d15ecfe2e-istio-podinfo" pod "58829723-0628-457e-80b8-f81d15ecfe2e" (UID: "58829723-0628-457e-80b8-f81d15ecfe2e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:09:00.463742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.453032 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/36f2d843-7d6a-4606-b6b3-6904a2b6cc29-istio-podinfo" pod "36f2d843-7d6a-4606-b6b3-6904a2b6cc29" (UID: "36f2d843-7d6a-4606-b6b3-6904a2b6cc29"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:09:00.466508 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.464346 4305 scope.go:122] "RemoveContainer" containerID="dfbf78fe82e05e2a90b38ad21dac6ac7e2793aa394447992c7651e802a95478d" Nov 03 21:09:00.466508 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:09:00.464740 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfbf78fe82e05e2a90b38ad21dac6ac7e2793aa394447992c7651e802a95478d\": container with ID starting with dfbf78fe82e05e2a90b38ad21dac6ac7e2793aa394447992c7651e802a95478d not found: ID does not exist" containerID="dfbf78fe82e05e2a90b38ad21dac6ac7e2793aa394447992c7651e802a95478d" Nov 03 21:09:00.466508 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.464776 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfbf78fe82e05e2a90b38ad21dac6ac7e2793aa394447992c7651e802a95478d"} err="failed to get container status \"dfbf78fe82e05e2a90b38ad21dac6ac7e2793aa394447992c7651e802a95478d\": rpc error: code = NotFound desc = could not find container \"dfbf78fe82e05e2a90b38ad21dac6ac7e2793aa394447992c7651e802a95478d\": container with ID starting with dfbf78fe82e05e2a90b38ad21dac6ac7e2793aa394447992c7651e802a95478d not found: ID does not exist" Nov 03 21:09:00.466508 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.464799 4305 scope.go:122] "RemoveContainer" containerID="ea7e5f654bccbde6382c9f018f3f2ec5f2bc78514d2af0f94c66559e0d9aae46" Nov 03 21:09:00.472291 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.472256 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ff154f8-a2c6-49a4-bbc8-f5026ec757bd-istio-podinfo" pod "0ff154f8-a2c6-49a4-bbc8-f5026ec757bd" (UID: "0ff154f8-a2c6-49a4-bbc8-f5026ec757bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:09:00.476764 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.476739 4305 scope.go:122] "RemoveContainer" containerID="ea7e5f654bccbde6382c9f018f3f2ec5f2bc78514d2af0f94c66559e0d9aae46" Nov 03 21:09:00.477133 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:09:00.477109 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea7e5f654bccbde6382c9f018f3f2ec5f2bc78514d2af0f94c66559e0d9aae46\": container with ID starting with ea7e5f654bccbde6382c9f018f3f2ec5f2bc78514d2af0f94c66559e0d9aae46 not found: ID does not exist" containerID="ea7e5f654bccbde6382c9f018f3f2ec5f2bc78514d2af0f94c66559e0d9aae46" Nov 03 21:09:00.477252 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.477141 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea7e5f654bccbde6382c9f018f3f2ec5f2bc78514d2af0f94c66559e0d9aae46"} err="failed to get container status \"ea7e5f654bccbde6382c9f018f3f2ec5f2bc78514d2af0f94c66559e0d9aae46\": rpc error: code = NotFound desc = could not find container \"ea7e5f654bccbde6382c9f018f3f2ec5f2bc78514d2af0f94c66559e0d9aae46\": container with ID starting with ea7e5f654bccbde6382c9f018f3f2ec5f2bc78514d2af0f94c66559e0d9aae46 not found: ID does not exist" Nov 03 21:09:00.477252 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.477162 4305 scope.go:122] "RemoveContainer" containerID="8818f76d7f9e22d97b1005dad732e19ae4011a562bc75443701eec18a1f9726e" Nov 03 21:09:00.480094 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.480058 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36f2d843-7d6a-4606-b6b3-6904a2b6cc29-istio-envoy" pod "36f2d843-7d6a-4606-b6b3-6904a2b6cc29" (UID: "36f2d843-7d6a-4606-b6b3-6904a2b6cc29"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:00.480094 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.480092 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36f2d843-7d6a-4606-b6b3-6904a2b6cc29-istio-token" pod "36f2d843-7d6a-4606-b6b3-6904a2b6cc29" (UID: "36f2d843-7d6a-4606-b6b3-6904a2b6cc29"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:09:00.481747 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.481725 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58829723-0628-457e-80b8-f81d15ecfe2e-istio-token" pod "58829723-0628-457e-80b8-f81d15ecfe2e" (UID: "58829723-0628-457e-80b8-f81d15ecfe2e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:09:00.482222 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.482196 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58829723-0628-457e-80b8-f81d15ecfe2e-kube-api-access-l9bmc" pod "58829723-0628-457e-80b8-f81d15ecfe2e" (UID: "58829723-0628-457e-80b8-f81d15ecfe2e"). InnerVolumeSpecName "kube-api-access-l9bmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:09:00.482335 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.482270 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ff154f8-a2c6-49a4-bbc8-f5026ec757bd-kube-api-access-6tkls" pod "0ff154f8-a2c6-49a4-bbc8-f5026ec757bd" (UID: "0ff154f8-a2c6-49a4-bbc8-f5026ec757bd"). InnerVolumeSpecName "kube-api-access-6tkls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:09:00.493970 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.493942 4305 scope.go:122] "RemoveContainer" containerID="8818f76d7f9e22d97b1005dad732e19ae4011a562bc75443701eec18a1f9726e" Nov 03 21:09:00.499100 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:09:00.498919 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8818f76d7f9e22d97b1005dad732e19ae4011a562bc75443701eec18a1f9726e\": container with ID starting with 8818f76d7f9e22d97b1005dad732e19ae4011a562bc75443701eec18a1f9726e not found: ID does not exist" containerID="8818f76d7f9e22d97b1005dad732e19ae4011a562bc75443701eec18a1f9726e" Nov 03 21:09:00.499100 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.498965 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8818f76d7f9e22d97b1005dad732e19ae4011a562bc75443701eec18a1f9726e"} err="failed to get container status \"8818f76d7f9e22d97b1005dad732e19ae4011a562bc75443701eec18a1f9726e\": rpc error: code = NotFound desc = could not find container \"8818f76d7f9e22d97b1005dad732e19ae4011a562bc75443701eec18a1f9726e\": container with ID starting with 8818f76d7f9e22d97b1005dad732e19ae4011a562bc75443701eec18a1f9726e not found: ID does not exist" Nov 03 21:09:00.499100 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.498992 4305 scope.go:122] "RemoveContainer" containerID="4d111a68c0e46f09480dfdfe969cb1010022c19b4dd169cff64e9ee661f2334c" Nov 03 21:09:00.528704 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.528659 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94bcfeb9-714d-4f25-8ce7-c6c4577ea837-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94bcfeb9-714d-4f25-8ce7-c6c4577ea837-workload-certs\") pod \"94bcfeb9-714d-4f25-8ce7-c6c4577ea837\" (UID: \"94bcfeb9-714d-4f25-8ce7-c6c4577ea837\") " Nov 03 21:09:00.528914 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.528734 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94bcfeb9-714d-4f25-8ce7-c6c4577ea837-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94bcfeb9-714d-4f25-8ce7-c6c4577ea837-istiod-ca-cert\") pod \"94bcfeb9-714d-4f25-8ce7-c6c4577ea837\" (UID: \"94bcfeb9-714d-4f25-8ce7-c6c4577ea837\") " Nov 03 21:09:00.528914 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.528761 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/94bcfeb9-714d-4f25-8ce7-c6c4577ea837-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94bcfeb9-714d-4f25-8ce7-c6c4577ea837-istio-podinfo\") pod \"94bcfeb9-714d-4f25-8ce7-c6c4577ea837\" (UID: \"94bcfeb9-714d-4f25-8ce7-c6c4577ea837\") " Nov 03 21:09:00.529030 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.528921 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94bcfeb9-714d-4f25-8ce7-c6c4577ea837-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94bcfeb9-714d-4f25-8ce7-c6c4577ea837-istio-envoy\") pod \"94bcfeb9-714d-4f25-8ce7-c6c4577ea837\" (UID: \"94bcfeb9-714d-4f25-8ce7-c6c4577ea837\") " Nov 03 21:09:00.529030 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.528958 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94bcfeb9-714d-4f25-8ce7-c6c4577ea837-workload-certs" pod "94bcfeb9-714d-4f25-8ce7-c6c4577ea837" (UID: "94bcfeb9-714d-4f25-8ce7-c6c4577ea837"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:00.529030 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.528969 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94bcfeb9-714d-4f25-8ce7-c6c4577ea837-kube-api-access-fm84w\" (UniqueName: \"kubernetes.io/projected/94bcfeb9-714d-4f25-8ce7-c6c4577ea837-kube-api-access-fm84w\") pod \"94bcfeb9-714d-4f25-8ce7-c6c4577ea837\" (UID: \"94bcfeb9-714d-4f25-8ce7-c6c4577ea837\") " Nov 03 21:09:00.529197 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.529123 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94bcfeb9-714d-4f25-8ce7-c6c4577ea837-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94bcfeb9-714d-4f25-8ce7-c6c4577ea837-credential-socket\") pod \"94bcfeb9-714d-4f25-8ce7-c6c4577ea837\" (UID: \"94bcfeb9-714d-4f25-8ce7-c6c4577ea837\") " Nov 03 21:09:00.529197 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.529123 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94bcfeb9-714d-4f25-8ce7-c6c4577ea837-istiod-ca-cert" pod "94bcfeb9-714d-4f25-8ce7-c6c4577ea837" (UID: "94bcfeb9-714d-4f25-8ce7-c6c4577ea837"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:09:00.529197 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.529162 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94bcfeb9-714d-4f25-8ce7-c6c4577ea837-istio-token\" (UniqueName: \"kubernetes.io/projected/94bcfeb9-714d-4f25-8ce7-c6c4577ea837-istio-token\") pod \"94bcfeb9-714d-4f25-8ce7-c6c4577ea837\" (UID: \"94bcfeb9-714d-4f25-8ce7-c6c4577ea837\") " Nov 03 21:09:00.529352 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.529237 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94bcfeb9-714d-4f25-8ce7-c6c4577ea837-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94bcfeb9-714d-4f25-8ce7-c6c4577ea837-istio-data\") pod \"94bcfeb9-714d-4f25-8ce7-c6c4577ea837\" (UID: \"94bcfeb9-714d-4f25-8ce7-c6c4577ea837\") " Nov 03 21:09:00.529352 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.529327 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94bcfeb9-714d-4f25-8ce7-c6c4577ea837-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94bcfeb9-714d-4f25-8ce7-c6c4577ea837-workload-socket\") pod \"94bcfeb9-714d-4f25-8ce7-c6c4577ea837\" (UID: \"94bcfeb9-714d-4f25-8ce7-c6c4577ea837\") " Nov 03 21:09:00.529471 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.529372 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94bcfeb9-714d-4f25-8ce7-c6c4577ea837-credential-socket" pod "94bcfeb9-714d-4f25-8ce7-c6c4577ea837" (UID: "94bcfeb9-714d-4f25-8ce7-c6c4577ea837"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:00.529584 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.529550 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94bcfeb9-714d-4f25-8ce7-c6c4577ea837-istio-data" pod "94bcfeb9-714d-4f25-8ce7-c6c4577ea837" (UID: "94bcfeb9-714d-4f25-8ce7-c6c4577ea837"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:00.529678 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.529657 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94bcfeb9-714d-4f25-8ce7-c6c4577ea837-workload-socket" pod "94bcfeb9-714d-4f25-8ce7-c6c4577ea837" (UID: "94bcfeb9-714d-4f25-8ce7-c6c4577ea837"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:00.530122 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.530101 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94bcfeb9-714d-4f25-8ce7-c6c4577ea837-workload-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:00.530122 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.530124 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36f2d843-7d6a-4606-b6b3-6904a2b6cc29-workload-certs\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:00.530312 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.530139 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9bmc\" (UniqueName: \"kubernetes.io/projected/58829723-0628-457e-80b8-f81d15ecfe2e-kube-api-access-l9bmc\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:00.530312 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.530153 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36f2d843-7d6a-4606-b6b3-6904a2b6cc29-istio-envoy\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:00.530312 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.530165 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94bcfeb9-714d-4f25-8ce7-c6c4577ea837-workload-certs\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:00.530312 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.530177 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58829723-0628-457e-80b8-f81d15ecfe2e-istio-data\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:00.530312 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.530190 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94bcfeb9-714d-4f25-8ce7-c6c4577ea837-istiod-ca-cert\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:00.530312 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.530204 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff154f8-a2c6-49a4-bbc8-f5026ec757bd-credential-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:00.530312 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.530217 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58829723-0628-457e-80b8-f81d15ecfe2e-istio-podinfo\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:00.530312 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.530231 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58829723-0628-457e-80b8-f81d15ecfe2e-istio-envoy\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:00.530312 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.530244 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36f2d843-7d6a-4606-b6b3-6904a2b6cc29-istio-data\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:00.530312 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.530258 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36f2d843-7d6a-4606-b6b3-6904a2b6cc29-istiod-ca-cert\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:00.530312 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.530274 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tkls\" (UniqueName: \"kubernetes.io/projected/0ff154f8-a2c6-49a4-bbc8-f5026ec757bd-kube-api-access-6tkls\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:00.530312 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.530288 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ff154f8-a2c6-49a4-bbc8-f5026ec757bd-istio-podinfo\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:00.530312 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.530302 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58829723-0628-457e-80b8-f81d15ecfe2e-workload-certs\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:00.530312 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.530315 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff154f8-a2c6-49a4-bbc8-f5026ec757bd-workload-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:00.531108 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.530328 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ff154f8-a2c6-49a4-bbc8-f5026ec757bd-istio-envoy\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:00.531108 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.530343 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94bcfeb9-714d-4f25-8ce7-c6c4577ea837-credential-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:00.531108 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.530357 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ff154f8-a2c6-49a4-bbc8-f5026ec757bd-workload-certs\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:00.531108 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.530371 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ff154f8-a2c6-49a4-bbc8-f5026ec757bd-istiod-ca-cert\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:00.531108 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.530384 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ff154f8-a2c6-49a4-bbc8-f5026ec757bd-istio-token\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:00.531108 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.530399 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36f2d843-7d6a-4606-b6b3-6904a2b6cc29-credential-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:00.531108 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.530413 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36f2d843-7d6a-4606-b6b3-6904a2b6cc29-istio-podinfo\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:00.531108 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.530428 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58829723-0628-457e-80b8-f81d15ecfe2e-istiod-ca-cert\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:00.531108 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.530461 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ff154f8-a2c6-49a4-bbc8-f5026ec757bd-istio-data\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:00.531108 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.530474 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tkw7\" (UniqueName: \"kubernetes.io/projected/36f2d843-7d6a-4606-b6b3-6904a2b6cc29-kube-api-access-2tkw7\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:00.531108 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.530488 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58829723-0628-457e-80b8-f81d15ecfe2e-istio-token\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:00.531108 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.530502 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94bcfeb9-714d-4f25-8ce7-c6c4577ea837-istio-data\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:00.531108 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.530529 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58829723-0628-457e-80b8-f81d15ecfe2e-workload-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:00.531108 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.530542 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36f2d843-7d6a-4606-b6b3-6904a2b6cc29-istio-token\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:00.531613 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.531585 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/94bcfeb9-714d-4f25-8ce7-c6c4577ea837-istio-podinfo" pod "94bcfeb9-714d-4f25-8ce7-c6c4577ea837" (UID: "94bcfeb9-714d-4f25-8ce7-c6c4577ea837"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:09:00.531785 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.531763 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94bcfeb9-714d-4f25-8ce7-c6c4577ea837-istio-token" pod "94bcfeb9-714d-4f25-8ce7-c6c4577ea837" (UID: "94bcfeb9-714d-4f25-8ce7-c6c4577ea837"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:09:00.532277 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.532249 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94bcfeb9-714d-4f25-8ce7-c6c4577ea837-istio-envoy" pod "94bcfeb9-714d-4f25-8ce7-c6c4577ea837" (UID: "94bcfeb9-714d-4f25-8ce7-c6c4577ea837"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:00.532277 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.532256 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94bcfeb9-714d-4f25-8ce7-c6c4577ea837-kube-api-access-fm84w" pod "94bcfeb9-714d-4f25-8ce7-c6c4577ea837" (UID: "94bcfeb9-714d-4f25-8ce7-c6c4577ea837"). InnerVolumeSpecName "kube-api-access-fm84w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:09:00.554724 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.554699 4305 scope.go:122] "RemoveContainer" containerID="4d111a68c0e46f09480dfdfe969cb1010022c19b4dd169cff64e9ee661f2334c" Nov 03 21:09:00.555120 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:09:00.555093 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d111a68c0e46f09480dfdfe969cb1010022c19b4dd169cff64e9ee661f2334c\": container with ID starting with 4d111a68c0e46f09480dfdfe969cb1010022c19b4dd169cff64e9ee661f2334c not found: ID does not exist" containerID="4d111a68c0e46f09480dfdfe969cb1010022c19b4dd169cff64e9ee661f2334c" Nov 03 21:09:00.555243 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.555132 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d111a68c0e46f09480dfdfe969cb1010022c19b4dd169cff64e9ee661f2334c"} err="failed to get container status \"4d111a68c0e46f09480dfdfe969cb1010022c19b4dd169cff64e9ee661f2334c\": rpc error: code = NotFound desc = could not find container \"4d111a68c0e46f09480dfdfe969cb1010022c19b4dd169cff64e9ee661f2334c\": container with ID starting with 4d111a68c0e46f09480dfdfe969cb1010022c19b4dd169cff64e9ee661f2334c not found: ID does not exist" Nov 03 21:09:00.555243 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.555151 4305 scope.go:122] "RemoveContainer" containerID="d950b9777f3fe72d899dfe98f108d37bb059466e0e8b4c1b5216870ecd4bafcc" Nov 03 21:09:00.571699 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.571672 4305 scope.go:122] "RemoveContainer" containerID="d950b9777f3fe72d899dfe98f108d37bb059466e0e8b4c1b5216870ecd4bafcc" Nov 03 21:09:00.572177 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:09:00.572140 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d950b9777f3fe72d899dfe98f108d37bb059466e0e8b4c1b5216870ecd4bafcc\": container with ID starting with d950b9777f3fe72d899dfe98f108d37bb059466e0e8b4c1b5216870ecd4bafcc not found: ID does not exist" containerID="d950b9777f3fe72d899dfe98f108d37bb059466e0e8b4c1b5216870ecd4bafcc" Nov 03 21:09:00.572287 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.572192 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d950b9777f3fe72d899dfe98f108d37bb059466e0e8b4c1b5216870ecd4bafcc"} err="failed to get container status \"d950b9777f3fe72d899dfe98f108d37bb059466e0e8b4c1b5216870ecd4bafcc\": rpc error: code = NotFound desc = could not find container \"d950b9777f3fe72d899dfe98f108d37bb059466e0e8b4c1b5216870ecd4bafcc\": container with ID starting with d950b9777f3fe72d899dfe98f108d37bb059466e0e8b4c1b5216870ecd4bafcc not found: ID does not exist" Nov 03 21:09:00.572287 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.572218 4305 scope.go:122] "RemoveContainer" containerID="b1008eb64b520f663b7e82c72eecf11ea5d8037cbd6c6f312cfe85e5a20e97b7" Nov 03 21:09:00.606316 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.606182 4305 scope.go:122] "RemoveContainer" containerID="b1008eb64b520f663b7e82c72eecf11ea5d8037cbd6c6f312cfe85e5a20e97b7" Nov 03 21:09:00.606664 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:09:00.606635 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1008eb64b520f663b7e82c72eecf11ea5d8037cbd6c6f312cfe85e5a20e97b7\": container with ID starting with b1008eb64b520f663b7e82c72eecf11ea5d8037cbd6c6f312cfe85e5a20e97b7 not found: ID does not exist" containerID="b1008eb64b520f663b7e82c72eecf11ea5d8037cbd6c6f312cfe85e5a20e97b7" Nov 03 21:09:00.606742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.606680 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1008eb64b520f663b7e82c72eecf11ea5d8037cbd6c6f312cfe85e5a20e97b7"} err="failed to get container status \"b1008eb64b520f663b7e82c72eecf11ea5d8037cbd6c6f312cfe85e5a20e97b7\": rpc error: code = NotFound desc = could not find container \"b1008eb64b520f663b7e82c72eecf11ea5d8037cbd6c6f312cfe85e5a20e97b7\": container with ID starting with b1008eb64b520f663b7e82c72eecf11ea5d8037cbd6c6f312cfe85e5a20e97b7 not found: ID does not exist" Nov 03 21:09:00.632033 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.631988 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94bcfeb9-714d-4f25-8ce7-c6c4577ea837-istio-token\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:00.632033 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.632033 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94bcfeb9-714d-4f25-8ce7-c6c4577ea837-istio-podinfo\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:00.632281 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.632049 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94bcfeb9-714d-4f25-8ce7-c6c4577ea837-istio-envoy\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:00.632281 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.632064 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm84w\" (UniqueName: \"kubernetes.io/projected/94bcfeb9-714d-4f25-8ce7-c6c4577ea837-kube-api-access-fm84w\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:00.851268 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.851231 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh"] Nov 03 21:09:00.855176 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.855142 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-c6vmh"] Nov 03 21:09:00.898062 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.898029 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1" path="/var/lib/kubelet/pods/17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1/volumes" Nov 03 21:09:00.968527 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:00.968471 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-346e532f-lztv8" podStartSLOduration=2.968451368 podStartE2EDuration="2.968451368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:08:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:09:00.966942182 +0000 UTC m=+1776.192262015" watchObservedRunningTime="2026-11-03 21:09:00.968451368 +0000 UTC m=+1776.193771158" Nov 03 21:09:01.191117 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:01.191007 4305 patch_prober.go:29] interesting pod/e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:09:01.191117 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:01.191077 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd" podUID="0ff154f8-a2c6-49a4-bbc8-f5026ec757bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:09:01.209660 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:01.209625 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x"] Nov 03 21:09:01.217545 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:01.217510 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-lrx9x"] Nov 03 21:09:01.326894 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:01.326859 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d"] Nov 03 21:09:01.330583 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:01.330555 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-ppj7d"] Nov 03 21:09:01.449908 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:01.449798 4305 generic.go:356] "Generic (PLEG): container finished" podID="99b7150f-e466-4734-97db-cde9002e38b0" containerID="4cfb4c063c1bbbe62e1ca1fc0192559c0707b52c482cb306758bd0397f8f5359" exitCode=0 Nov 03 21:09:01.449908 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:01.449885 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-346e532f-lztv8" event={"ID":"99b7150f-e466-4734-97db-cde9002e38b0","Type":"ContainerDied","Data":"4cfb4c063c1bbbe62e1ca1fc0192559c0707b52c482cb306758bd0397f8f5359"} Nov 03 21:09:01.637187 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:01.637156 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb6523bb-dd4qf" Nov 03 21:09:01.751204 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:01.751110 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fb6d58d-39c4-4f67-876e-7000d5833d82-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fb6d58d-39c4-4f67-876e-7000d5833d82-maas-api-openshift-ca\") pod \"6fb6d58d-39c4-4f67-876e-7000d5833d82\" (UID: \"6fb6d58d-39c4-4f67-876e-7000d5833d82\") " Nov 03 21:09:01.751377 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:01.751319 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fb6d58d-39c4-4f67-876e-7000d5833d82-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fb6d58d-39c4-4f67-876e-7000d5833d82-maas-api-xks-ca\") pod \"6fb6d58d-39c4-4f67-876e-7000d5833d82\" (UID: \"6fb6d58d-39c4-4f67-876e-7000d5833d82\") " Nov 03 21:09:01.751547 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:01.751532 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fb6d58d-39c4-4f67-876e-7000d5833d82-kube-api-access-fjqc5\" (UniqueName: \"kubernetes.io/projected/6fb6d58d-39c4-4f67-876e-7000d5833d82-kube-api-access-fjqc5\") pod \"6fb6d58d-39c4-4f67-876e-7000d5833d82\" (UID: \"6fb6d58d-39c4-4f67-876e-7000d5833d82\") " Nov 03 21:09:01.751631 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:01.751610 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fb6d58d-39c4-4f67-876e-7000d5833d82-maas-api-openshift-ca" pod "6fb6d58d-39c4-4f67-876e-7000d5833d82" (UID: "6fb6d58d-39c4-4f67-876e-7000d5833d82"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:09:01.752355 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:01.752327 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fb6d58d-39c4-4f67-876e-7000d5833d82-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:01.753859 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:01.753834 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fb6d58d-39c4-4f67-876e-7000d5833d82-maas-api-xks-ca" pod "6fb6d58d-39c4-4f67-876e-7000d5833d82" (UID: "6fb6d58d-39c4-4f67-876e-7000d5833d82"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:09:01.753961 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:01.753934 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fb6d58d-39c4-4f67-876e-7000d5833d82-kube-api-access-fjqc5" pod "6fb6d58d-39c4-4f67-876e-7000d5833d82" (UID: "6fb6d58d-39c4-4f67-876e-7000d5833d82"). InnerVolumeSpecName "kube-api-access-fjqc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:09:01.802804 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:01.802767 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28"] Nov 03 21:09:01.814480 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:01.814425 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-l8v28"] Nov 03 21:09:01.854216 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:01.854184 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fb6d58d-39c4-4f67-876e-7000d5833d82-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:01.854216 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:01.854214 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjqc5\" (UniqueName: \"kubernetes.io/projected/6fb6d58d-39c4-4f67-876e-7000d5833d82-kube-api-access-fjqc5\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:01.925244 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:01.925206 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd"] Nov 03 21:09:01.929732 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:01.929696 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-499187-openshift-default-c6598c9fc-rl5vd"] Nov 03 21:09:02.158321 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:02.158275 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd"] Nov 03 21:09:02.165965 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:02.165926 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-5wtfd"] Nov 03 21:09:02.278860 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:02.278825 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx"] Nov 03 21:09:02.297562 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:02.297526 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-6c6kx"] Nov 03 21:09:02.461879 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:02.461799 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb6523bb-dd4qf" Nov 03 21:09:02.462016 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:02.461804 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb6523bb-dd4qf" event={"ID":"6fb6d58d-39c4-4f67-876e-7000d5833d82","Type":"ContainerDied","Data":"c3dd604355a771a30ae42e2c3d68f64a6fae3d359f49542a72b47163da3bbe02"} Nov 03 21:09:02.462016 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:02.461975 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3dd604355a771a30ae42e2c3d68f64a6fae3d359f49542a72b47163da3bbe02" Nov 03 21:09:02.634769 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:02.634742 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-346e532f-lztv8" Nov 03 21:09:02.665716 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:02.665680 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99b7150f-e466-4734-97db-cde9002e38b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99b7150f-e466-4734-97db-cde9002e38b0-maas-api-xks-ca\") pod \"99b7150f-e466-4734-97db-cde9002e38b0\" (UID: \"99b7150f-e466-4734-97db-cde9002e38b0\") " Nov 03 21:09:02.665873 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:02.665810 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99b7150f-e466-4734-97db-cde9002e38b0-kube-api-access-zcxv7\" (UniqueName: \"kubernetes.io/projected/99b7150f-e466-4734-97db-cde9002e38b0-kube-api-access-zcxv7\") pod \"99b7150f-e466-4734-97db-cde9002e38b0\" (UID: \"99b7150f-e466-4734-97db-cde9002e38b0\") " Nov 03 21:09:02.665873 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:02.665840 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99b7150f-e466-4734-97db-cde9002e38b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99b7150f-e466-4734-97db-cde9002e38b0-maas-api-openshift-ca\") pod \"99b7150f-e466-4734-97db-cde9002e38b0\" (UID: \"99b7150f-e466-4734-97db-cde9002e38b0\") " Nov 03 21:09:02.666282 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:02.666251 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99b7150f-e466-4734-97db-cde9002e38b0-maas-api-openshift-ca" pod "99b7150f-e466-4734-97db-cde9002e38b0" (UID: "99b7150f-e466-4734-97db-cde9002e38b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:09:02.668055 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:02.668024 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99b7150f-e466-4734-97db-cde9002e38b0-maas-api-xks-ca" pod "99b7150f-e466-4734-97db-cde9002e38b0" (UID: "99b7150f-e466-4734-97db-cde9002e38b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:09:02.668148 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:02.668087 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99b7150f-e466-4734-97db-cde9002e38b0-kube-api-access-zcxv7" pod "99b7150f-e466-4734-97db-cde9002e38b0" (UID: "99b7150f-e466-4734-97db-cde9002e38b0"). InnerVolumeSpecName "kube-api-access-zcxv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:09:02.767167 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:02.767085 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcxv7\" (UniqueName: \"kubernetes.io/projected/99b7150f-e466-4734-97db-cde9002e38b0-kube-api-access-zcxv7\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:02.767366 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:02.767346 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99b7150f-e466-4734-97db-cde9002e38b0-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:02.767366 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:02.767365 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99b7150f-e466-4734-97db-cde9002e38b0-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:02.907600 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:02.907559 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ff154f8-a2c6-49a4-bbc8-f5026ec757bd" path="/var/lib/kubelet/pods/0ff154f8-a2c6-49a4-bbc8-f5026ec757bd/volumes" Nov 03 21:09:02.908215 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:02.908189 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1920ffc0-36e7-4d99-9410-444ad24afd9c" path="/var/lib/kubelet/pods/1920ffc0-36e7-4d99-9410-444ad24afd9c/volumes" Nov 03 21:09:02.908985 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:02.908956 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36f2d843-7d6a-4606-b6b3-6904a2b6cc29" path="/var/lib/kubelet/pods/36f2d843-7d6a-4606-b6b3-6904a2b6cc29/volumes" Nov 03 21:09:02.910190 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:02.910164 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58829723-0628-457e-80b8-f81d15ecfe2e" path="/var/lib/kubelet/pods/58829723-0628-457e-80b8-f81d15ecfe2e/volumes" Nov 03 21:09:02.910837 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:02.910813 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94bcfeb9-714d-4f25-8ce7-c6c4577ea837" path="/var/lib/kubelet/pods/94bcfeb9-714d-4f25-8ce7-c6c4577ea837/volumes" Nov 03 21:09:02.911361 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:02.911341 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9103e89-52f2-4022-bd9b-dc554c82f479" path="/var/lib/kubelet/pods/d9103e89-52f2-4022-bd9b-dc554c82f479/volumes" Nov 03 21:09:03.475490 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:03.475461 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-346e532f-lztv8" Nov 03 21:09:03.475937 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:03.475459 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-346e532f-lztv8" event={"ID":"99b7150f-e466-4734-97db-cde9002e38b0","Type":"ContainerDied","Data":"c4bd7a64d5b291a7bb43a010eedccd27f4adc3b8c7489aa98736bc561de21593"} Nov 03 21:09:03.475937 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:03.475597 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4bd7a64d5b291a7bb43a010eedccd27f4adc3b8c7489aa98736bc561de21593" Nov 03 21:09:16.325736 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:09:16.325686 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:09:18.187138 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:18.187102 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ca4642-6dsn2"] Nov 03 21:09:18.187838 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:18.187820 4305 cpu_manager.go:414] "containerMap: removing container" podUID="36f2d843-7d6a-4606-b6b3-6904a2b6cc29" containerName="istio-proxy" Nov 03 21:09:18.187888 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:18.187839 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36f2d843-7d6a-4606-b6b3-6904a2b6cc29" containerName="istio-proxy" Nov 03 21:09:18.187888 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:18.187852 4305 cpu_manager.go:414] "containerMap: removing container" podUID="94bcfeb9-714d-4f25-8ce7-c6c4577ea837" containerName="istio-proxy" Nov 03 21:09:18.187888 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:18.187858 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94bcfeb9-714d-4f25-8ce7-c6c4577ea837" containerName="istio-proxy" Nov 03 21:09:18.187888 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:18.187866 4305 cpu_manager.go:414] "containerMap: removing container" podUID="58829723-0628-457e-80b8-f81d15ecfe2e" containerName="istio-proxy" Nov 03 21:09:18.187888 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:18.187871 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58829723-0628-457e-80b8-f81d15ecfe2e" containerName="istio-proxy" Nov 03 21:09:18.187888 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:18.187886 4305 cpu_manager.go:414] "containerMap: removing container" podUID="17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1" containerName="istio-proxy" Nov 03 21:09:18.188153 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:18.187891 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1" containerName="istio-proxy" Nov 03 21:09:18.188153 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:18.187911 4305 cpu_manager.go:414] "containerMap: removing container" podUID="0ff154f8-a2c6-49a4-bbc8-f5026ec757bd" containerName="istio-proxy" Nov 03 21:09:18.188153 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:18.187916 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ff154f8-a2c6-49a4-bbc8-f5026ec757bd" containerName="istio-proxy" Nov 03 21:09:18.188153 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:18.187958 4305 cpu_manager.go:414] "containerMap: removing container" podUID="6fb6d58d-39c4-4f67-876e-7000d5833d82" containerName="revoke-keys" Nov 03 21:09:18.188153 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:18.187963 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fb6d58d-39c4-4f67-876e-7000d5833d82" containerName="revoke-keys" Nov 03 21:09:18.188153 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:18.187973 4305 cpu_manager.go:414] "containerMap: removing container" podUID="58829723-0628-457e-80b8-f81d15ecfe2e" containerName="istio-proxy" Nov 03 21:09:18.188153 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:18.187978 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58829723-0628-457e-80b8-f81d15ecfe2e" containerName="istio-proxy" Nov 03 21:09:18.188153 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:18.187990 4305 cpu_manager.go:414] "containerMap: removing container" podUID="1920ffc0-36e7-4d99-9410-444ad24afd9c" containerName="istio-proxy" Nov 03 21:09:18.188153 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:18.187994 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1920ffc0-36e7-4d99-9410-444ad24afd9c" containerName="istio-proxy" Nov 03 21:09:18.188153 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:18.188003 4305 cpu_manager.go:414] "containerMap: removing container" podUID="d9103e89-52f2-4022-bd9b-dc554c82f479" containerName="istio-proxy" Nov 03 21:09:18.188153 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:18.188007 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9103e89-52f2-4022-bd9b-dc554c82f479" containerName="istio-proxy" Nov 03 21:09:18.188153 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:18.188024 4305 cpu_manager.go:414] "containerMap: removing container" podUID="99b7150f-e466-4734-97db-cde9002e38b0" containerName="revoke-keys" Nov 03 21:09:18.188153 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:18.188031 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99b7150f-e466-4734-97db-cde9002e38b0" containerName="revoke-keys" Nov 03 21:09:18.188153 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:18.188159 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="58829723-0628-457e-80b8-f81d15ecfe2e" containerName="istio-proxy" Nov 03 21:09:18.188651 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:18.188172 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="1920ffc0-36e7-4d99-9410-444ad24afd9c" containerName="istio-proxy" Nov 03 21:09:18.188651 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:18.188184 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9103e89-52f2-4022-bd9b-dc554c82f479" containerName="istio-proxy" Nov 03 21:09:18.188651 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:18.188198 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="99b7150f-e466-4734-97db-cde9002e38b0" containerName="revoke-keys" Nov 03 21:09:18.188651 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:18.188213 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="36f2d843-7d6a-4606-b6b3-6904a2b6cc29" containerName="istio-proxy" Nov 03 21:09:18.188651 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:18.188219 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="94bcfeb9-714d-4f25-8ce7-c6c4577ea837" containerName="istio-proxy" Nov 03 21:09:18.188651 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:18.188231 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="17bbc6a8-9ea0-4b75-a88c-b15525fb0dc1" containerName="istio-proxy" Nov 03 21:09:18.188651 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:18.188240 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ff154f8-a2c6-49a4-bbc8-f5026ec757bd" containerName="istio-proxy" Nov 03 21:09:18.188651 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:18.188246 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fb6d58d-39c4-4f67-876e-7000d5833d82" containerName="revoke-keys" Nov 03 21:09:18.191943 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:18.191923 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ca4642-6dsn2" Nov 03 21:09:18.247801 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:18.247768 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ca4642-6dsn2"] Nov 03 21:09:18.253268 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:18.253234 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77vtd\"" Nov 03 21:09:18.292784 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:18.292746 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:09:18.292784 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:18.292743 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:09:18.330093 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:18.330050 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phrcc\" (UniqueName: \"kubernetes.io/projected/559f3f98-6c70-42b2-aa59-22cb1902e41d-kube-api-access-phrcc\") pod \"maas-api-revoke-keys-e2e-worker-w4-ca4642-6dsn2\" (UID: \"559f3f98-6c70-42b2-aa59-22cb1902e41d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ca4642-6dsn2" Nov 03 21:09:18.330300 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:18.330198 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/559f3f98-6c70-42b2-aa59-22cb1902e41d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-ca4642-6dsn2\" (UID: \"559f3f98-6c70-42b2-aa59-22cb1902e41d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ca4642-6dsn2" Nov 03 21:09:18.330495 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:18.330406 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/559f3f98-6c70-42b2-aa59-22cb1902e41d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-ca4642-6dsn2\" (UID: \"559f3f98-6c70-42b2-aa59-22cb1902e41d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ca4642-6dsn2" Nov 03 21:09:18.351936 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:18.351889 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:09:18.432425 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:18.432383 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/559f3f98-6c70-42b2-aa59-22cb1902e41d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-ca4642-6dsn2\" (UID: \"559f3f98-6c70-42b2-aa59-22cb1902e41d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ca4642-6dsn2" Nov 03 21:09:18.432666 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:18.432546 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phrcc\" (UniqueName: \"kubernetes.io/projected/559f3f98-6c70-42b2-aa59-22cb1902e41d-kube-api-access-phrcc\") pod \"maas-api-revoke-keys-e2e-worker-w4-ca4642-6dsn2\" (UID: \"559f3f98-6c70-42b2-aa59-22cb1902e41d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ca4642-6dsn2" Nov 03 21:09:18.432666 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:18.432651 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/559f3f98-6c70-42b2-aa59-22cb1902e41d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-ca4642-6dsn2\" (UID: \"559f3f98-6c70-42b2-aa59-22cb1902e41d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ca4642-6dsn2" Nov 03 21:09:18.433251 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:18.433230 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/559f3f98-6c70-42b2-aa59-22cb1902e41d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-ca4642-6dsn2\" (UID: \"559f3f98-6c70-42b2-aa59-22cb1902e41d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ca4642-6dsn2" Nov 03 21:09:18.435005 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:18.434983 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/559f3f98-6c70-42b2-aa59-22cb1902e41d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-ca4642-6dsn2\" (UID: \"559f3f98-6c70-42b2-aa59-22cb1902e41d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ca4642-6dsn2" Nov 03 21:09:18.493746 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:18.493671 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phrcc\" (UniqueName: \"kubernetes.io/projected/559f3f98-6c70-42b2-aa59-22cb1902e41d-kube-api-access-phrcc\") pod \"maas-api-revoke-keys-e2e-worker-w4-ca4642-6dsn2\" (UID: \"559f3f98-6c70-42b2-aa59-22cb1902e41d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ca4642-6dsn2" Nov 03 21:09:18.506624 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:18.506594 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ca4642-6dsn2" Nov 03 21:09:18.897377 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:18.897337 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ca4642-6dsn2"] Nov 03 21:09:18.898428 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:09:18.898398 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod559f3f98_6c70_42b2_aa59_22cb1902e41d.slice/crio-9f58f59cf55ad9a17eb0986904dad89fadc3928d76e85f42e69a8ae38bee5b92 WatchSource:0}: Error finding container 9f58f59cf55ad9a17eb0986904dad89fadc3928d76e85f42e69a8ae38bee5b92: Status 404 returned error can't find the container with id 9f58f59cf55ad9a17eb0986904dad89fadc3928d76e85f42e69a8ae38bee5b92 Nov 03 21:09:19.624768 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:19.624658 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ca4642-6dsn2" event={"ID":"559f3f98-6c70-42b2-aa59-22cb1902e41d","Type":"ContainerStarted","Data":"464727cb6a9016bb6dba67b58aa975a8adc5acffd3bd585ff676b7c56a3388f8"} Nov 03 21:09:19.624768 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:19.624752 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ca4642-6dsn2" event={"ID":"559f3f98-6c70-42b2-aa59-22cb1902e41d","Type":"ContainerStarted","Data":"9f58f59cf55ad9a17eb0986904dad89fadc3928d76e85f42e69a8ae38bee5b92"} Nov 03 21:09:19.754345 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:19.754279 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ca4642-6dsn2" podStartSLOduration=1.754260183 podStartE2EDuration="1.754260183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:09:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:09:19.749089102 +0000 UTC m=+1794.974408918" watchObservedRunningTime="2026-11-03 21:09:19.754260183 +0000 UTC m=+1794.979579974" Nov 03 21:09:20.633898 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:20.633863 4305 generic.go:356] "Generic (PLEG): container finished" podID="559f3f98-6c70-42b2-aa59-22cb1902e41d" containerID="464727cb6a9016bb6dba67b58aa975a8adc5acffd3bd585ff676b7c56a3388f8" exitCode=0 Nov 03 21:09:20.634317 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:20.633948 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ca4642-6dsn2" event={"ID":"559f3f98-6c70-42b2-aa59-22cb1902e41d","Type":"ContainerDied","Data":"464727cb6a9016bb6dba67b58aa975a8adc5acffd3bd585ff676b7c56a3388f8"} Nov 03 21:09:21.804830 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:21.804802 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ca4642-6dsn2" Nov 03 21:09:21.982574 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:21.982537 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/559f3f98-6c70-42b2-aa59-22cb1902e41d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/559f3f98-6c70-42b2-aa59-22cb1902e41d-maas-api-xks-ca\") pod \"559f3f98-6c70-42b2-aa59-22cb1902e41d\" (UID: \"559f3f98-6c70-42b2-aa59-22cb1902e41d\") " Nov 03 21:09:21.982783 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:21.982701 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/559f3f98-6c70-42b2-aa59-22cb1902e41d-kube-api-access-phrcc\" (UniqueName: \"kubernetes.io/projected/559f3f98-6c70-42b2-aa59-22cb1902e41d-kube-api-access-phrcc\") pod \"559f3f98-6c70-42b2-aa59-22cb1902e41d\" (UID: \"559f3f98-6c70-42b2-aa59-22cb1902e41d\") " Nov 03 21:09:21.982855 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:21.982838 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/559f3f98-6c70-42b2-aa59-22cb1902e41d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/559f3f98-6c70-42b2-aa59-22cb1902e41d-maas-api-openshift-ca\") pod \"559f3f98-6c70-42b2-aa59-22cb1902e41d\" (UID: \"559f3f98-6c70-42b2-aa59-22cb1902e41d\") " Nov 03 21:09:21.983156 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:21.983131 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/559f3f98-6c70-42b2-aa59-22cb1902e41d-maas-api-openshift-ca" pod "559f3f98-6c70-42b2-aa59-22cb1902e41d" (UID: "559f3f98-6c70-42b2-aa59-22cb1902e41d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:09:21.983461 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:21.983423 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/559f3f98-6c70-42b2-aa59-22cb1902e41d-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:21.984982 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:21.984956 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/559f3f98-6c70-42b2-aa59-22cb1902e41d-maas-api-xks-ca" pod "559f3f98-6c70-42b2-aa59-22cb1902e41d" (UID: "559f3f98-6c70-42b2-aa59-22cb1902e41d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:09:21.985061 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:21.985024 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/559f3f98-6c70-42b2-aa59-22cb1902e41d-kube-api-access-phrcc" pod "559f3f98-6c70-42b2-aa59-22cb1902e41d" (UID: "559f3f98-6c70-42b2-aa59-22cb1902e41d"). InnerVolumeSpecName "kube-api-access-phrcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:09:22.085418 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:22.085374 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phrcc\" (UniqueName: \"kubernetes.io/projected/559f3f98-6c70-42b2-aa59-22cb1902e41d-kube-api-access-phrcc\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:22.085418 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:22.085412 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/559f3f98-6c70-42b2-aa59-22cb1902e41d-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:22.650824 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:22.650791 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ca4642-6dsn2" Nov 03 21:09:22.650824 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:22.650812 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ca4642-6dsn2" event={"ID":"559f3f98-6c70-42b2-aa59-22cb1902e41d","Type":"ContainerDied","Data":"9f58f59cf55ad9a17eb0986904dad89fadc3928d76e85f42e69a8ae38bee5b92"} Nov 03 21:09:22.651049 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:22.650866 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f58f59cf55ad9a17eb0986904dad89fadc3928d76e85f42e69a8ae38bee5b92" Nov 03 21:09:22.658107 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:22.658074 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4"] Nov 03 21:09:22.707538 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:22.707501 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc"] Nov 03 21:09:29.958852 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:29.958821 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ca4642-56449c8659-vtm4r"] Nov 03 21:09:30.554054 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:30.554022 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-499187-vqvzq"] Nov 03 21:09:30.554697 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:30.554681 4305 cpu_manager.go:414] "containerMap: removing container" podUID="559f3f98-6c70-42b2-aa59-22cb1902e41d" containerName="revoke-keys" Nov 03 21:09:30.554751 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:30.554697 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="559f3f98-6c70-42b2-aa59-22cb1902e41d" containerName="revoke-keys" Nov 03 21:09:30.554835 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:30.554826 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="58829723-0628-457e-80b8-f81d15ecfe2e" containerName="istio-proxy" Nov 03 21:09:30.554876 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:30.554840 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="559f3f98-6c70-42b2-aa59-22cb1902e41d" containerName="revoke-keys" Nov 03 21:09:30.558749 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:30.558723 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-499187-vqvzq" Nov 03 21:09:30.611384 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:30.611351 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:09:30.611750 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:30.611679 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:09:30.611750 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:30.611690 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:09:30.611750 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:30.611749 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77vtd\"" Nov 03 21:09:30.616110 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:30.616082 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-499187-vqvzq"] Nov 03 21:09:30.657870 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:30.657826 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/772927b9-7b5b-4dc7-b62e-c5dd61b54960-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-499187-vqvzq\" (UID: \"772927b9-7b5b-4dc7-b62e-c5dd61b54960\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-499187-vqvzq" Nov 03 21:09:30.658076 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:30.657907 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/772927b9-7b5b-4dc7-b62e-c5dd61b54960-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-499187-vqvzq\" (UID: \"772927b9-7b5b-4dc7-b62e-c5dd61b54960\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-499187-vqvzq" Nov 03 21:09:30.658159 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:30.658073 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9wmf\" (UniqueName: \"kubernetes.io/projected/772927b9-7b5b-4dc7-b62e-c5dd61b54960-kube-api-access-s9wmf\") pod \"maas-api-revoke-keys-e2e-worker-w0-499187-vqvzq\" (UID: \"772927b9-7b5b-4dc7-b62e-c5dd61b54960\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-499187-vqvzq" Nov 03 21:09:30.759523 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:30.759486 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/772927b9-7b5b-4dc7-b62e-c5dd61b54960-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-499187-vqvzq\" (UID: \"772927b9-7b5b-4dc7-b62e-c5dd61b54960\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-499187-vqvzq" Nov 03 21:09:30.759710 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:30.759542 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/772927b9-7b5b-4dc7-b62e-c5dd61b54960-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-499187-vqvzq\" (UID: \"772927b9-7b5b-4dc7-b62e-c5dd61b54960\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-499187-vqvzq" Nov 03 21:09:30.759710 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:30.759600 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9wmf\" (UniqueName: \"kubernetes.io/projected/772927b9-7b5b-4dc7-b62e-c5dd61b54960-kube-api-access-s9wmf\") pod \"maas-api-revoke-keys-e2e-worker-w0-499187-vqvzq\" (UID: \"772927b9-7b5b-4dc7-b62e-c5dd61b54960\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-499187-vqvzq" Nov 03 21:09:30.760155 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:30.760132 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/772927b9-7b5b-4dc7-b62e-c5dd61b54960-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-499187-vqvzq\" (UID: \"772927b9-7b5b-4dc7-b62e-c5dd61b54960\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-499187-vqvzq" Nov 03 21:09:30.762241 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:30.762207 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/772927b9-7b5b-4dc7-b62e-c5dd61b54960-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-499187-vqvzq\" (UID: \"772927b9-7b5b-4dc7-b62e-c5dd61b54960\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-499187-vqvzq" Nov 03 21:09:30.829014 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:30.828939 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9wmf\" (UniqueName: \"kubernetes.io/projected/772927b9-7b5b-4dc7-b62e-c5dd61b54960-kube-api-access-s9wmf\") pod \"maas-api-revoke-keys-e2e-worker-w0-499187-vqvzq\" (UID: \"772927b9-7b5b-4dc7-b62e-c5dd61b54960\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-499187-vqvzq" Nov 03 21:09:30.873882 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:30.873838 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-499187-vqvzq" Nov 03 21:09:31.203110 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:31.203074 4305 scope.go:122] "RemoveContainer" containerID="0701da96404727ac9cb151aadbc88365711690aa75fdfd45676d67aafd67386f" Nov 03 21:09:31.214160 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:31.214131 4305 scope.go:122] "RemoveContainer" containerID="38d7749aaa68308fd0a5db54d28456f32902deb2f2ffff6199326705de114ca8" Nov 03 21:09:31.225203 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:31.225178 4305 scope.go:122] "RemoveContainer" containerID="4966a8b3aabf843074be11bd2088ce6affa5d3dac9edebad7bf98b2d1ccd0352" Nov 03 21:09:31.236016 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:31.235995 4305 scope.go:122] "RemoveContainer" containerID="d02f9ebaf3fb8ca133e33c3a2ca5a2c7933aea686a87a2ec2e2eae119da51e57" Nov 03 21:09:31.247469 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:31.247448 4305 scope.go:122] "RemoveContainer" containerID="b1f5fcca462a110a31f89fdad970f1c3c4cd83f7ed7e0e477fbabc4903226b93" Nov 03 21:09:31.262893 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:31.262858 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-499187-vqvzq"] Nov 03 21:09:31.266030 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:09:31.266000 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod772927b9_7b5b_4dc7_b62e_c5dd61b54960.slice/crio-ab134a03c6fffebfc23cd1220bd6147102c804786bd37d897cf6a43dc25758e5 WatchSource:0}: Error finding container ab134a03c6fffebfc23cd1220bd6147102c804786bd37d897cf6a43dc25758e5: Status 404 returned error can't find the container with id ab134a03c6fffebfc23cd1220bd6147102c804786bd37d897cf6a43dc25758e5 Nov 03 21:09:31.730033 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:31.730000 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-499187-vqvzq" event={"ID":"772927b9-7b5b-4dc7-b62e-c5dd61b54960","Type":"ContainerStarted","Data":"ab134a03c6fffebfc23cd1220bd6147102c804786bd37d897cf6a43dc25758e5"} Nov 03 21:09:32.738613 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:32.738566 4305 generic.go:356] "Generic (PLEG): container finished" podID="772927b9-7b5b-4dc7-b62e-c5dd61b54960" containerID="eccac510d12dbcac88497133af76a034cffcd5a96cb39a51281ea3e9e759515a" exitCode=0 Nov 03 21:09:32.739088 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:32.738652 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-499187-vqvzq" event={"ID":"772927b9-7b5b-4dc7-b62e-c5dd61b54960","Type":"ContainerDied","Data":"eccac510d12dbcac88497133af76a034cffcd5a96cb39a51281ea3e9e759515a"} Nov 03 21:09:33.928857 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:33.928824 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-499187-vqvzq" Nov 03 21:09:33.934306 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:33.934278 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/772927b9-7b5b-4dc7-b62e-c5dd61b54960-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/772927b9-7b5b-4dc7-b62e-c5dd61b54960-maas-api-xks-ca\") pod \"772927b9-7b5b-4dc7-b62e-c5dd61b54960\" (UID: \"772927b9-7b5b-4dc7-b62e-c5dd61b54960\") " Nov 03 21:09:33.934523 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:33.934378 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/772927b9-7b5b-4dc7-b62e-c5dd61b54960-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/772927b9-7b5b-4dc7-b62e-c5dd61b54960-maas-api-openshift-ca\") pod \"772927b9-7b5b-4dc7-b62e-c5dd61b54960\" (UID: \"772927b9-7b5b-4dc7-b62e-c5dd61b54960\") " Nov 03 21:09:33.934571 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:33.934542 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/772927b9-7b5b-4dc7-b62e-c5dd61b54960-kube-api-access-s9wmf\" (UniqueName: \"kubernetes.io/projected/772927b9-7b5b-4dc7-b62e-c5dd61b54960-kube-api-access-s9wmf\") pod \"772927b9-7b5b-4dc7-b62e-c5dd61b54960\" (UID: \"772927b9-7b5b-4dc7-b62e-c5dd61b54960\") " Nov 03 21:09:33.934811 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:33.934782 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/772927b9-7b5b-4dc7-b62e-c5dd61b54960-maas-api-openshift-ca" pod "772927b9-7b5b-4dc7-b62e-c5dd61b54960" (UID: "772927b9-7b5b-4dc7-b62e-c5dd61b54960"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:09:33.935329 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:33.935303 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/772927b9-7b5b-4dc7-b62e-c5dd61b54960-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:33.936708 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:33.936684 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/772927b9-7b5b-4dc7-b62e-c5dd61b54960-maas-api-xks-ca" pod "772927b9-7b5b-4dc7-b62e-c5dd61b54960" (UID: "772927b9-7b5b-4dc7-b62e-c5dd61b54960"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:09:33.936825 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:33.936803 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/772927b9-7b5b-4dc7-b62e-c5dd61b54960-kube-api-access-s9wmf" pod "772927b9-7b5b-4dc7-b62e-c5dd61b54960" (UID: "772927b9-7b5b-4dc7-b62e-c5dd61b54960"). InnerVolumeSpecName "kube-api-access-s9wmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:09:34.036479 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:34.036342 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/772927b9-7b5b-4dc7-b62e-c5dd61b54960-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:34.036479 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:34.036379 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9wmf\" (UniqueName: \"kubernetes.io/projected/772927b9-7b5b-4dc7-b62e-c5dd61b54960-kube-api-access-s9wmf\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:34.760574 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:34.760541 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-499187-vqvzq" Nov 03 21:09:34.760796 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:34.760575 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-499187-vqvzq" event={"ID":"772927b9-7b5b-4dc7-b62e-c5dd61b54960","Type":"ContainerDied","Data":"ab134a03c6fffebfc23cd1220bd6147102c804786bd37d897cf6a43dc25758e5"} Nov 03 21:09:34.760796 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:34.760634 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab134a03c6fffebfc23cd1220bd6147102c804786bd37d897cf6a43dc25758e5" Nov 03 21:09:34.914705 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:34.914668 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr"] Nov 03 21:09:34.946378 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:34.946312 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8"] Nov 03 21:09:34.959974 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:34.959924 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ca4642-56449c8659-vtm4r" podUID="6f951e66-9baa-4ad3-994f-fa1b2e0d4828" containerName="payload-pre-processing" containerID="cri-o://4e05564f2fc434141b76871a062356686d18f7075903c6117bc78779d72f6c64" gracePeriod=25 Nov 03 21:09:37.680400 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:37.680337 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" podUID="be06bc96-808a-46e6-9872-01baf1a11a8d" containerName="main" containerID="cri-o://ece75c0b5cd7e0b25238ecbce9b0648aac6247002dfbfd6590167e97b7b6791e" gracePeriod=45 Nov 03 21:09:37.729897 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:37.729857 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" podUID="80de7944-70f6-49e1-a0e8-4b82d5caad0d" containerName="main" containerID="cri-o://47f02a5ff2acc2619ef94f3e74a994157a324827f21ffd93323181a5d3576771" gracePeriod=45 Nov 03 21:09:37.823128 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:37.823085 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" podUID="80de7944-70f6-49e1-a0e8-4b82d5caad0d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": dial tcp 10.134.0.59:8000: connect: connection refused" Nov 03 21:09:37.824394 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:37.824367 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" podUID="be06bc96-808a-46e6-9872-01baf1a11a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.107:8000/ready\": dial tcp 10.134.0.107:8000: connect: connection refused" Nov 03 21:09:38.028598 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:38.028252 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" Nov 03 21:09:38.057944 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:38.057918 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" Nov 03 21:09:38.103128 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:38.103082 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be06bc96-808a-46e6-9872-01baf1a11a8d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be06bc96-808a-46e6-9872-01baf1a11a8d-tmp-dir\") pod \"be06bc96-808a-46e6-9872-01baf1a11a8d\" (UID: \"be06bc96-808a-46e6-9872-01baf1a11a8d\") " Nov 03 21:09:38.103326 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:38.103139 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80de7944-70f6-49e1-a0e8-4b82d5caad0d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/80de7944-70f6-49e1-a0e8-4b82d5caad0d-dshm\") pod \"80de7944-70f6-49e1-a0e8-4b82d5caad0d\" (UID: \"80de7944-70f6-49e1-a0e8-4b82d5caad0d\") " Nov 03 21:09:38.103326 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:38.103207 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80de7944-70f6-49e1-a0e8-4b82d5caad0d-kube-api-access-5lxcn\" (UniqueName: \"kubernetes.io/projected/80de7944-70f6-49e1-a0e8-4b82d5caad0d-kube-api-access-5lxcn\") pod \"80de7944-70f6-49e1-a0e8-4b82d5caad0d\" (UID: \"80de7944-70f6-49e1-a0e8-4b82d5caad0d\") " Nov 03 21:09:38.103326 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:38.103310 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be06bc96-808a-46e6-9872-01baf1a11a8d-kube-api-access-v8v4r\" (UniqueName: \"kubernetes.io/projected/be06bc96-808a-46e6-9872-01baf1a11a8d-kube-api-access-v8v4r\") pod \"be06bc96-808a-46e6-9872-01baf1a11a8d\" (UID: \"be06bc96-808a-46e6-9872-01baf1a11a8d\") " Nov 03 21:09:38.103580 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:38.103386 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be06bc96-808a-46e6-9872-01baf1a11a8d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be06bc96-808a-46e6-9872-01baf1a11a8d-model-cache\") pod \"be06bc96-808a-46e6-9872-01baf1a11a8d\" (UID: \"be06bc96-808a-46e6-9872-01baf1a11a8d\") " Nov 03 21:09:38.103580 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:38.103410 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be06bc96-808a-46e6-9872-01baf1a11a8d-tmp-dir" pod "be06bc96-808a-46e6-9872-01baf1a11a8d" (UID: "be06bc96-808a-46e6-9872-01baf1a11a8d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:38.103580 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:38.103425 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80de7944-70f6-49e1-a0e8-4b82d5caad0d-home\" (UniqueName: \"kubernetes.io/empty-dir/80de7944-70f6-49e1-a0e8-4b82d5caad0d-home\") pod \"80de7944-70f6-49e1-a0e8-4b82d5caad0d\" (UID: \"80de7944-70f6-49e1-a0e8-4b82d5caad0d\") " Nov 03 21:09:38.103580 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:38.103559 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80de7944-70f6-49e1-a0e8-4b82d5caad0d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80de7944-70f6-49e1-a0e8-4b82d5caad0d-model-cache\") pod \"80de7944-70f6-49e1-a0e8-4b82d5caad0d\" (UID: \"80de7944-70f6-49e1-a0e8-4b82d5caad0d\") " Nov 03 21:09:38.103790 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:38.103594 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be06bc96-808a-46e6-9872-01baf1a11a8d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/be06bc96-808a-46e6-9872-01baf1a11a8d-dshm\") pod \"be06bc96-808a-46e6-9872-01baf1a11a8d\" (UID: \"be06bc96-808a-46e6-9872-01baf1a11a8d\") " Nov 03 21:09:38.103790 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:38.103658 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80de7944-70f6-49e1-a0e8-4b82d5caad0d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80de7944-70f6-49e1-a0e8-4b82d5caad0d-tmp-dir\") pod \"80de7944-70f6-49e1-a0e8-4b82d5caad0d\" (UID: \"80de7944-70f6-49e1-a0e8-4b82d5caad0d\") " Nov 03 21:09:38.103790 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:38.103727 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be06bc96-808a-46e6-9872-01baf1a11a8d-tls-certs\" (UniqueName: \"kubernetes.io/secret/be06bc96-808a-46e6-9872-01baf1a11a8d-tls-certs\") pod \"be06bc96-808a-46e6-9872-01baf1a11a8d\" (UID: \"be06bc96-808a-46e6-9872-01baf1a11a8d\") " Nov 03 21:09:38.103790 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:38.103762 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be06bc96-808a-46e6-9872-01baf1a11a8d-home\" (UniqueName: \"kubernetes.io/empty-dir/be06bc96-808a-46e6-9872-01baf1a11a8d-home\") pod \"be06bc96-808a-46e6-9872-01baf1a11a8d\" (UID: \"be06bc96-808a-46e6-9872-01baf1a11a8d\") " Nov 03 21:09:38.104008 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:38.103824 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80de7944-70f6-49e1-a0e8-4b82d5caad0d-home" pod "80de7944-70f6-49e1-a0e8-4b82d5caad0d" (UID: "80de7944-70f6-49e1-a0e8-4b82d5caad0d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:38.104008 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:38.103834 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80de7944-70f6-49e1-a0e8-4b82d5caad0d-tls-certs\" (UniqueName: \"kubernetes.io/secret/80de7944-70f6-49e1-a0e8-4b82d5caad0d-tls-certs\") pod \"80de7944-70f6-49e1-a0e8-4b82d5caad0d\" (UID: \"80de7944-70f6-49e1-a0e8-4b82d5caad0d\") " Nov 03 21:09:38.104008 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:38.103994 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80de7944-70f6-49e1-a0e8-4b82d5caad0d-model-cache" pod "80de7944-70f6-49e1-a0e8-4b82d5caad0d" (UID: "80de7944-70f6-49e1-a0e8-4b82d5caad0d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:38.104155 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:38.104031 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be06bc96-808a-46e6-9872-01baf1a11a8d-model-cache" pod "be06bc96-808a-46e6-9872-01baf1a11a8d" (UID: "be06bc96-808a-46e6-9872-01baf1a11a8d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:38.104247 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:38.104223 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80de7944-70f6-49e1-a0e8-4b82d5caad0d-tmp-dir" pod "80de7944-70f6-49e1-a0e8-4b82d5caad0d" (UID: "80de7944-70f6-49e1-a0e8-4b82d5caad0d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:38.104638 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:38.104396 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be06bc96-808a-46e6-9872-01baf1a11a8d-home" pod "be06bc96-808a-46e6-9872-01baf1a11a8d" (UID: "be06bc96-808a-46e6-9872-01baf1a11a8d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:38.104762 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:38.104666 4305 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80de7944-70f6-49e1-a0e8-4b82d5caad0d-home\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:38.104762 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:38.104688 4305 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80de7944-70f6-49e1-a0e8-4b82d5caad0d-model-cache\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:38.104762 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:38.104704 4305 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80de7944-70f6-49e1-a0e8-4b82d5caad0d-tmp-dir\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:38.104762 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:38.104718 4305 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be06bc96-808a-46e6-9872-01baf1a11a8d-home\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:38.104762 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:38.104730 4305 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be06bc96-808a-46e6-9872-01baf1a11a8d-tmp-dir\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:38.104762 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:38.104743 4305 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be06bc96-808a-46e6-9872-01baf1a11a8d-model-cache\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:38.106477 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:38.106451 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be06bc96-808a-46e6-9872-01baf1a11a8d-tls-certs" pod "be06bc96-808a-46e6-9872-01baf1a11a8d" (UID: "be06bc96-808a-46e6-9872-01baf1a11a8d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:09:38.106575 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:38.106493 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80de7944-70f6-49e1-a0e8-4b82d5caad0d-dshm" pod "80de7944-70f6-49e1-a0e8-4b82d5caad0d" (UID: "80de7944-70f6-49e1-a0e8-4b82d5caad0d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:38.106575 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:38.106568 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80de7944-70f6-49e1-a0e8-4b82d5caad0d-tls-certs" pod "80de7944-70f6-49e1-a0e8-4b82d5caad0d" (UID: "80de7944-70f6-49e1-a0e8-4b82d5caad0d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:09:38.106691 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:38.106591 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be06bc96-808a-46e6-9872-01baf1a11a8d-kube-api-access-v8v4r" pod "be06bc96-808a-46e6-9872-01baf1a11a8d" (UID: "be06bc96-808a-46e6-9872-01baf1a11a8d"). InnerVolumeSpecName "kube-api-access-v8v4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:09:38.106828 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:38.106806 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be06bc96-808a-46e6-9872-01baf1a11a8d-dshm" pod "be06bc96-808a-46e6-9872-01baf1a11a8d" (UID: "be06bc96-808a-46e6-9872-01baf1a11a8d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:38.107185 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:38.107162 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80de7944-70f6-49e1-a0e8-4b82d5caad0d-kube-api-access-5lxcn" pod "80de7944-70f6-49e1-a0e8-4b82d5caad0d" (UID: "80de7944-70f6-49e1-a0e8-4b82d5caad0d"). InnerVolumeSpecName "kube-api-access-5lxcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:09:38.205831 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:38.205740 4305 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be06bc96-808a-46e6-9872-01baf1a11a8d-dshm\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:38.205831 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:38.205768 4305 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be06bc96-808a-46e6-9872-01baf1a11a8d-tls-certs\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:38.205831 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:38.205778 4305 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80de7944-70f6-49e1-a0e8-4b82d5caad0d-tls-certs\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:38.205831 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:38.205787 4305 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80de7944-70f6-49e1-a0e8-4b82d5caad0d-dshm\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:38.205831 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:38.205796 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lxcn\" (UniqueName: \"kubernetes.io/projected/80de7944-70f6-49e1-a0e8-4b82d5caad0d-kube-api-access-5lxcn\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:38.205831 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:38.205805 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8v4r\" (UniqueName: \"kubernetes.io/projected/be06bc96-808a-46e6-9872-01baf1a11a8d-kube-api-access-v8v4r\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:38.794583 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:38.794545 4305 generic.go:356] "Generic (PLEG): container finished" podID="be06bc96-808a-46e6-9872-01baf1a11a8d" containerID="ece75c0b5cd7e0b25238ecbce9b0648aac6247002dfbfd6590167e97b7b6791e" exitCode=0 Nov 03 21:09:38.795232 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:38.795204 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" Nov 03 21:09:38.795587 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:38.795111 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" event={"ID":"be06bc96-808a-46e6-9872-01baf1a11a8d","Type":"ContainerDied","Data":"ece75c0b5cd7e0b25238ecbce9b0648aac6247002dfbfd6590167e97b7b6791e"} Nov 03 21:09:38.795708 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:38.795618 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4" event={"ID":"be06bc96-808a-46e6-9872-01baf1a11a8d","Type":"ContainerDied","Data":"f5db05a87ceedb9042a912e51d8fb8b358c7f3b6572e3c9387b7b599e9ccb56e"} Nov 03 21:09:38.795708 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:38.795650 4305 scope.go:122] "RemoveContainer" containerID="ece75c0b5cd7e0b25238ecbce9b0648aac6247002dfbfd6590167e97b7b6791e" Nov 03 21:09:38.798255 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:38.798225 4305 generic.go:356] "Generic (PLEG): container finished" podID="80de7944-70f6-49e1-a0e8-4b82d5caad0d" containerID="47f02a5ff2acc2619ef94f3e74a994157a324827f21ffd93323181a5d3576771" exitCode=0 Nov 03 21:09:38.798494 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:38.798424 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" event={"ID":"80de7944-70f6-49e1-a0e8-4b82d5caad0d","Type":"ContainerDied","Data":"47f02a5ff2acc2619ef94f3e74a994157a324827f21ffd93323181a5d3576771"} Nov 03 21:09:38.798585 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:38.798517 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" event={"ID":"80de7944-70f6-49e1-a0e8-4b82d5caad0d","Type":"ContainerDied","Data":"ee2293057a73bfdec3efd737f07315a1dc55a78c74dbcf1a4db0f40ec467af96"} Nov 03 21:09:38.798676 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:38.798596 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc" Nov 03 21:09:38.810727 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:38.810708 4305 scope.go:122] "RemoveContainer" containerID="ece75c0b5cd7e0b25238ecbce9b0648aac6247002dfbfd6590167e97b7b6791e" Nov 03 21:09:38.811040 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:09:38.811018 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ece75c0b5cd7e0b25238ecbce9b0648aac6247002dfbfd6590167e97b7b6791e\": container with ID starting with ece75c0b5cd7e0b25238ecbce9b0648aac6247002dfbfd6590167e97b7b6791e not found: ID does not exist" containerID="ece75c0b5cd7e0b25238ecbce9b0648aac6247002dfbfd6590167e97b7b6791e" Nov 03 21:09:38.811135 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:38.811052 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ece75c0b5cd7e0b25238ecbce9b0648aac6247002dfbfd6590167e97b7b6791e"} err="failed to get container status \"ece75c0b5cd7e0b25238ecbce9b0648aac6247002dfbfd6590167e97b7b6791e\": rpc error: code = NotFound desc = could not find container \"ece75c0b5cd7e0b25238ecbce9b0648aac6247002dfbfd6590167e97b7b6791e\": container with ID starting with ece75c0b5cd7e0b25238ecbce9b0648aac6247002dfbfd6590167e97b7b6791e not found: ID does not exist" Nov 03 21:09:38.811135 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:38.811076 4305 scope.go:122] "RemoveContainer" containerID="47f02a5ff2acc2619ef94f3e74a994157a324827f21ffd93323181a5d3576771" Nov 03 21:09:38.821711 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:38.821687 4305 scope.go:122] "RemoveContainer" containerID="47f02a5ff2acc2619ef94f3e74a994157a324827f21ffd93323181a5d3576771" Nov 03 21:09:38.822096 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:09:38.822072 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47f02a5ff2acc2619ef94f3e74a994157a324827f21ffd93323181a5d3576771\": container with ID starting with 47f02a5ff2acc2619ef94f3e74a994157a324827f21ffd93323181a5d3576771 not found: ID does not exist" containerID="47f02a5ff2acc2619ef94f3e74a994157a324827f21ffd93323181a5d3576771" Nov 03 21:09:38.822215 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:38.822102 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47f02a5ff2acc2619ef94f3e74a994157a324827f21ffd93323181a5d3576771"} err="failed to get container status \"47f02a5ff2acc2619ef94f3e74a994157a324827f21ffd93323181a5d3576771\": rpc error: code = NotFound desc = could not find container \"47f02a5ff2acc2619ef94f3e74a994157a324827f21ffd93323181a5d3576771\": container with ID starting with 47f02a5ff2acc2619ef94f3e74a994157a324827f21ffd93323181a5d3576771 not found: ID does not exist" Nov 03 21:09:38.971024 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:38.970912 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc"] Nov 03 21:09:39.020226 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:39.020188 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-ca4642/simulator-w4-ca4642-kserve-6f45856c59-2xbqc"] Nov 03 21:09:39.085183 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:39.085147 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4"] Nov 03 21:09:39.089331 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:39.089301 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-ca4642/distinct-2-w4-ca4642-kserve-579f9b64f8-vkgq4"] Nov 03 21:09:40.955267 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:40.955220 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80de7944-70f6-49e1-a0e8-4b82d5caad0d" path="/var/lib/kubelet/pods/80de7944-70f6-49e1-a0e8-4b82d5caad0d/volumes" Nov 03 21:09:40.956153 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:40.956128 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be06bc96-808a-46e6-9872-01baf1a11a8d" path="/var/lib/kubelet/pods/be06bc96-808a-46e6-9872-01baf1a11a8d/volumes" Nov 03 21:09:43.147094 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:43.147055 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ca4642-56449c8659-vtm4r" podUID="6f951e66-9baa-4ad3-994f-fa1b2e0d4828" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:09:44.148842 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:44.148803 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6349b785-88d6n"] Nov 03 21:09:44.149476 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:44.149450 4305 cpu_manager.go:414] "containerMap: removing container" podUID="772927b9-7b5b-4dc7-b62e-c5dd61b54960" containerName="revoke-keys" Nov 03 21:09:44.149476 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:44.149468 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="772927b9-7b5b-4dc7-b62e-c5dd61b54960" containerName="revoke-keys" Nov 03 21:09:44.149577 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:44.149502 4305 cpu_manager.go:414] "containerMap: removing container" podUID="80de7944-70f6-49e1-a0e8-4b82d5caad0d" containerName="main" Nov 03 21:09:44.149577 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:44.149507 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80de7944-70f6-49e1-a0e8-4b82d5caad0d" containerName="main" Nov 03 21:09:44.149577 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:44.149531 4305 cpu_manager.go:414] "containerMap: removing container" podUID="be06bc96-808a-46e6-9872-01baf1a11a8d" containerName="main" Nov 03 21:09:44.149577 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:44.149536 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be06bc96-808a-46e6-9872-01baf1a11a8d" containerName="main" Nov 03 21:09:44.149718 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:44.149626 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="80de7944-70f6-49e1-a0e8-4b82d5caad0d" containerName="main" Nov 03 21:09:44.149718 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:44.149637 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="be06bc96-808a-46e6-9872-01baf1a11a8d" containerName="main" Nov 03 21:09:44.149718 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:44.149646 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="772927b9-7b5b-4dc7-b62e-c5dd61b54960" containerName="revoke-keys" Nov 03 21:09:44.154620 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:44.154598 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6349b785-88d6n" Nov 03 21:09:44.156335 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:44.156313 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5769a2cf-rmwbr"] Nov 03 21:09:44.160620 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:44.160601 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5769a2cf-rmwbr" Nov 03 21:09:44.191538 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:44.191496 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szm5r\" (UniqueName: \"kubernetes.io/projected/c31b220a-debb-4e97-9dd9-03c304faf447-kube-api-access-szm5r\") pod \"maas-api-revoke-sub-6349b785-88d6n\" (UID: \"c31b220a-debb-4e97-9dd9-03c304faf447\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6349b785-88d6n" Nov 03 21:09:44.191735 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:44.191668 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c31b220a-debb-4e97-9dd9-03c304faf447-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6349b785-88d6n\" (UID: \"c31b220a-debb-4e97-9dd9-03c304faf447\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6349b785-88d6n" Nov 03 21:09:44.191735 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:44.191711 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c31b220a-debb-4e97-9dd9-03c304faf447-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6349b785-88d6n\" (UID: \"c31b220a-debb-4e97-9dd9-03c304faf447\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6349b785-88d6n" Nov 03 21:09:44.206695 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:44.206668 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:09:44.206903 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:44.206797 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77vtd\"" Nov 03 21:09:44.206903 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:44.206851 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:09:44.206903 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:44.206880 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:09:44.262263 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:44.262216 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6349b785-88d6n"] Nov 03 21:09:44.266840 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:44.266809 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5769a2cf-rmwbr"] Nov 03 21:09:44.293734 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:44.293694 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h49x\" (UniqueName: \"kubernetes.io/projected/c6f56404-e1ad-4521-b8ba-ab22614197a0-kube-api-access-8h49x\") pod \"maas-api-revoke-sub-5769a2cf-rmwbr\" (UID: \"c6f56404-e1ad-4521-b8ba-ab22614197a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5769a2cf-rmwbr" Nov 03 21:09:44.293930 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:44.293844 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szm5r\" (UniqueName: \"kubernetes.io/projected/c31b220a-debb-4e97-9dd9-03c304faf447-kube-api-access-szm5r\") pod \"maas-api-revoke-sub-6349b785-88d6n\" (UID: \"c31b220a-debb-4e97-9dd9-03c304faf447\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6349b785-88d6n" Nov 03 21:09:44.293930 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:44.293893 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6f56404-e1ad-4521-b8ba-ab22614197a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5769a2cf-rmwbr\" (UID: \"c6f56404-e1ad-4521-b8ba-ab22614197a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5769a2cf-rmwbr" Nov 03 21:09:44.294033 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:44.293967 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c31b220a-debb-4e97-9dd9-03c304faf447-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6349b785-88d6n\" (UID: \"c31b220a-debb-4e97-9dd9-03c304faf447\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6349b785-88d6n" Nov 03 21:09:44.294033 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:44.293989 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6f56404-e1ad-4521-b8ba-ab22614197a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5769a2cf-rmwbr\" (UID: \"c6f56404-e1ad-4521-b8ba-ab22614197a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5769a2cf-rmwbr" Nov 03 21:09:44.294124 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:44.294081 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c31b220a-debb-4e97-9dd9-03c304faf447-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6349b785-88d6n\" (UID: \"c31b220a-debb-4e97-9dd9-03c304faf447\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6349b785-88d6n" Nov 03 21:09:44.294741 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:44.294711 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c31b220a-debb-4e97-9dd9-03c304faf447-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6349b785-88d6n\" (UID: \"c31b220a-debb-4e97-9dd9-03c304faf447\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6349b785-88d6n" Nov 03 21:09:44.296686 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:44.296664 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c31b220a-debb-4e97-9dd9-03c304faf447-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6349b785-88d6n\" (UID: \"c31b220a-debb-4e97-9dd9-03c304faf447\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6349b785-88d6n" Nov 03 21:09:44.355161 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:44.355120 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szm5r\" (UniqueName: \"kubernetes.io/projected/c31b220a-debb-4e97-9dd9-03c304faf447-kube-api-access-szm5r\") pod \"maas-api-revoke-sub-6349b785-88d6n\" (UID: \"c31b220a-debb-4e97-9dd9-03c304faf447\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6349b785-88d6n" Nov 03 21:09:44.395328 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:44.395290 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h49x\" (UniqueName: \"kubernetes.io/projected/c6f56404-e1ad-4521-b8ba-ab22614197a0-kube-api-access-8h49x\") pod \"maas-api-revoke-sub-5769a2cf-rmwbr\" (UID: \"c6f56404-e1ad-4521-b8ba-ab22614197a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5769a2cf-rmwbr" Nov 03 21:09:44.395574 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:44.395416 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6f56404-e1ad-4521-b8ba-ab22614197a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5769a2cf-rmwbr\" (UID: \"c6f56404-e1ad-4521-b8ba-ab22614197a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5769a2cf-rmwbr" Nov 03 21:09:44.395574 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:44.395515 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6f56404-e1ad-4521-b8ba-ab22614197a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5769a2cf-rmwbr\" (UID: \"c6f56404-e1ad-4521-b8ba-ab22614197a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5769a2cf-rmwbr" Nov 03 21:09:44.396185 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:44.396143 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6f56404-e1ad-4521-b8ba-ab22614197a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5769a2cf-rmwbr\" (UID: \"c6f56404-e1ad-4521-b8ba-ab22614197a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5769a2cf-rmwbr" Nov 03 21:09:44.398000 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:44.397980 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6f56404-e1ad-4521-b8ba-ab22614197a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5769a2cf-rmwbr\" (UID: \"c6f56404-e1ad-4521-b8ba-ab22614197a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5769a2cf-rmwbr" Nov 03 21:09:44.456858 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:44.456787 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h49x\" (UniqueName: \"kubernetes.io/projected/c6f56404-e1ad-4521-b8ba-ab22614197a0-kube-api-access-8h49x\") pod \"maas-api-revoke-sub-5769a2cf-rmwbr\" (UID: \"c6f56404-e1ad-4521-b8ba-ab22614197a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5769a2cf-rmwbr" Nov 03 21:09:44.470750 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:44.470721 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6349b785-88d6n" Nov 03 21:09:44.478172 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:44.478147 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5769a2cf-rmwbr" Nov 03 21:09:44.934282 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:44.934256 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6349b785-88d6n"] Nov 03 21:09:44.936158 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:09:44.936132 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc31b220a_debb_4e97_9dd9_03c304faf447.slice/crio-20f116e485d99d58aba57492affd1498287ac7f73b1472ac5ea05d29115bce15 WatchSource:0}: Error finding container 20f116e485d99d58aba57492affd1498287ac7f73b1472ac5ea05d29115bce15: Status 404 returned error can't find the container with id 20f116e485d99d58aba57492affd1498287ac7f73b1472ac5ea05d29115bce15 Nov 03 21:09:44.985916 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:44.985885 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5769a2cf-rmwbr"] Nov 03 21:09:44.986501 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:09:44.986472 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6f56404_e1ad_4521_b8ba_ab22614197a0.slice/crio-ea92c956cd437304efeab9d5f1d0e189858b39d3d943e07effa71680588fe5c9 WatchSource:0}: Error finding container ea92c956cd437304efeab9d5f1d0e189858b39d3d943e07effa71680588fe5c9: Status 404 returned error can't find the container with id ea92c956cd437304efeab9d5f1d0e189858b39d3d943e07effa71680588fe5c9 Nov 03 21:09:45.404591 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:45.404554 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-499187-55f6b9c44c-mmhjm"] Nov 03 21:09:45.874351 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:45.874313 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6349b785-88d6n" event={"ID":"c31b220a-debb-4e97-9dd9-03c304faf447","Type":"ContainerStarted","Data":"759b16b59e1faab5e6782884565e483c016e3b3b41a8b68c131ba3753a7fa2d7"} Nov 03 21:09:45.874610 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:45.874376 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6349b785-88d6n" event={"ID":"c31b220a-debb-4e97-9dd9-03c304faf447","Type":"ContainerStarted","Data":"20f116e485d99d58aba57492affd1498287ac7f73b1472ac5ea05d29115bce15"} Nov 03 21:09:45.876054 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:45.876024 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5769a2cf-rmwbr" event={"ID":"c6f56404-e1ad-4521-b8ba-ab22614197a0","Type":"ContainerStarted","Data":"3ff1b9893ad058ade0ef7e8fd58b613703fd0d7835d9d13ce9f7fce445abaaeb"} Nov 03 21:09:45.876215 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:45.876083 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5769a2cf-rmwbr" event={"ID":"c6f56404-e1ad-4521-b8ba-ab22614197a0","Type":"ContainerStarted","Data":"ea92c956cd437304efeab9d5f1d0e189858b39d3d943e07effa71680588fe5c9"} Nov 03 21:09:46.015128 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:46.015057 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6349b785-88d6n" podStartSLOduration=2.015038641 podStartE2EDuration="2.015038641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:09:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:09:46.010485048 +0000 UTC m=+1821.235804860" watchObservedRunningTime="2026-11-03 21:09:46.015038641 +0000 UTC m=+1821.240358433" Nov 03 21:09:46.132656 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:46.132603 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5769a2cf-rmwbr" podStartSLOduration=2.13258631 podStartE2EDuration="2.13258631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:09:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:09:46.129330282 +0000 UTC m=+1821.354650106" watchObservedRunningTime="2026-11-03 21:09:46.13258631 +0000 UTC m=+1821.357906100" Nov 03 21:09:46.324029 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:09:46.323987 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:09:46.901468 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:46.894092 4305 generic.go:356] "Generic (PLEG): container finished" podID="c31b220a-debb-4e97-9dd9-03c304faf447" containerID="759b16b59e1faab5e6782884565e483c016e3b3b41a8b68c131ba3753a7fa2d7" exitCode=0 Nov 03 21:09:46.923465 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:46.917578 4305 generic.go:356] "Generic (PLEG): container finished" podID="c6f56404-e1ad-4521-b8ba-ab22614197a0" containerID="3ff1b9893ad058ade0ef7e8fd58b613703fd0d7835d9d13ce9f7fce445abaaeb" exitCode=0 Nov 03 21:09:46.923465 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:46.918224 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6349b785-88d6n" event={"ID":"c31b220a-debb-4e97-9dd9-03c304faf447","Type":"ContainerDied","Data":"759b16b59e1faab5e6782884565e483c016e3b3b41a8b68c131ba3753a7fa2d7"} Nov 03 21:09:46.923465 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:46.918277 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5769a2cf-rmwbr" event={"ID":"c6f56404-e1ad-4521-b8ba-ab22614197a0","Type":"ContainerDied","Data":"3ff1b9893ad058ade0ef7e8fd58b613703fd0d7835d9d13ce9f7fce445abaaeb"} Nov 03 21:09:48.139268 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:48.139239 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6349b785-88d6n" Nov 03 21:09:48.143121 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:48.143093 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5769a2cf-rmwbr" Nov 03 21:09:48.283075 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:48.282977 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6f56404-e1ad-4521-b8ba-ab22614197a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6f56404-e1ad-4521-b8ba-ab22614197a0-maas-api-xks-ca\") pod \"c6f56404-e1ad-4521-b8ba-ab22614197a0\" (UID: \"c6f56404-e1ad-4521-b8ba-ab22614197a0\") " Nov 03 21:09:48.283266 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:48.283120 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c31b220a-debb-4e97-9dd9-03c304faf447-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c31b220a-debb-4e97-9dd9-03c304faf447-maas-api-openshift-ca\") pod \"c31b220a-debb-4e97-9dd9-03c304faf447\" (UID: \"c31b220a-debb-4e97-9dd9-03c304faf447\") " Nov 03 21:09:48.283266 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:48.283244 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c31b220a-debb-4e97-9dd9-03c304faf447-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c31b220a-debb-4e97-9dd9-03c304faf447-maas-api-xks-ca\") pod \"c31b220a-debb-4e97-9dd9-03c304faf447\" (UID: \"c31b220a-debb-4e97-9dd9-03c304faf447\") " Nov 03 21:09:48.283389 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:48.283331 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c31b220a-debb-4e97-9dd9-03c304faf447-kube-api-access-szm5r\" (UniqueName: \"kubernetes.io/projected/c31b220a-debb-4e97-9dd9-03c304faf447-kube-api-access-szm5r\") pod \"c31b220a-debb-4e97-9dd9-03c304faf447\" (UID: \"c31b220a-debb-4e97-9dd9-03c304faf447\") " Nov 03 21:09:48.283389 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:48.283371 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6f56404-e1ad-4521-b8ba-ab22614197a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6f56404-e1ad-4521-b8ba-ab22614197a0-maas-api-openshift-ca\") pod \"c6f56404-e1ad-4521-b8ba-ab22614197a0\" (UID: \"c6f56404-e1ad-4521-b8ba-ab22614197a0\") " Nov 03 21:09:48.283602 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:48.283406 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6f56404-e1ad-4521-b8ba-ab22614197a0-kube-api-access-8h49x\" (UniqueName: \"kubernetes.io/projected/c6f56404-e1ad-4521-b8ba-ab22614197a0-kube-api-access-8h49x\") pod \"c6f56404-e1ad-4521-b8ba-ab22614197a0\" (UID: \"c6f56404-e1ad-4521-b8ba-ab22614197a0\") " Nov 03 21:09:48.283959 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:48.283874 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c31b220a-debb-4e97-9dd9-03c304faf447-maas-api-openshift-ca" pod "c31b220a-debb-4e97-9dd9-03c304faf447" (UID: "c31b220a-debb-4e97-9dd9-03c304faf447"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:09:48.284156 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:48.284131 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c31b220a-debb-4e97-9dd9-03c304faf447-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:48.286117 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:48.286086 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6f56404-e1ad-4521-b8ba-ab22614197a0-maas-api-openshift-ca" pod "c6f56404-e1ad-4521-b8ba-ab22614197a0" (UID: "c6f56404-e1ad-4521-b8ba-ab22614197a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:09:48.286623 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:48.286313 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6f56404-e1ad-4521-b8ba-ab22614197a0-maas-api-xks-ca" pod "c6f56404-e1ad-4521-b8ba-ab22614197a0" (UID: "c6f56404-e1ad-4521-b8ba-ab22614197a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:09:48.286764 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:48.286738 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6f56404-e1ad-4521-b8ba-ab22614197a0-kube-api-access-8h49x" pod "c6f56404-e1ad-4521-b8ba-ab22614197a0" (UID: "c6f56404-e1ad-4521-b8ba-ab22614197a0"). InnerVolumeSpecName "kube-api-access-8h49x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:09:48.287575 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:48.287548 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c31b220a-debb-4e97-9dd9-03c304faf447-maas-api-xks-ca" pod "c31b220a-debb-4e97-9dd9-03c304faf447" (UID: "c31b220a-debb-4e97-9dd9-03c304faf447"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:09:48.287677 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:48.287648 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c31b220a-debb-4e97-9dd9-03c304faf447-kube-api-access-szm5r" pod "c31b220a-debb-4e97-9dd9-03c304faf447" (UID: "c31b220a-debb-4e97-9dd9-03c304faf447"). InnerVolumeSpecName "kube-api-access-szm5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:09:48.386061 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:48.386022 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szm5r\" (UniqueName: \"kubernetes.io/projected/c31b220a-debb-4e97-9dd9-03c304faf447-kube-api-access-szm5r\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:48.386061 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:48.386064 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6f56404-e1ad-4521-b8ba-ab22614197a0-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:48.386272 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:48.386080 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8h49x\" (UniqueName: \"kubernetes.io/projected/c6f56404-e1ad-4521-b8ba-ab22614197a0-kube-api-access-8h49x\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:48.386272 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:48.386094 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6f56404-e1ad-4521-b8ba-ab22614197a0-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:48.386272 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:48.386110 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c31b220a-debb-4e97-9dd9-03c304faf447-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:48.934629 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:48.934601 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6349b785-88d6n" Nov 03 21:09:48.934629 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:48.934616 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6349b785-88d6n" event={"ID":"c31b220a-debb-4e97-9dd9-03c304faf447","Type":"ContainerDied","Data":"20f116e485d99d58aba57492affd1498287ac7f73b1472ac5ea05d29115bce15"} Nov 03 21:09:48.934884 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:48.934664 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20f116e485d99d58aba57492affd1498287ac7f73b1472ac5ea05d29115bce15" Nov 03 21:09:48.936155 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:48.936134 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5769a2cf-rmwbr" Nov 03 21:09:48.936341 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:48.936135 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5769a2cf-rmwbr" event={"ID":"c6f56404-e1ad-4521-b8ba-ab22614197a0","Type":"ContainerDied","Data":"ea92c956cd437304efeab9d5f1d0e189858b39d3d943e07effa71680588fe5c9"} Nov 03 21:09:48.936341 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:48.936231 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea92c956cd437304efeab9d5f1d0e189858b39d3d943e07effa71680588fe5c9" Nov 03 21:09:49.958459 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:49.954001 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" podUID="8ee1f028-8503-4034-9954-eb06fcf402e7" containerName="main" containerID="cri-o://af86062f6aee09ba31fc2e3ec73039f924e12d291f1051c73faa10d4218e16c6" gracePeriod=45 Nov 03 21:09:49.976097 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:49.976040 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:09:50.015464 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:49.997874 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" podUID="793b8ca3-3a2e-49a1-9be4-4971ba59df09" containerName="main" containerID="cri-o://2b093b23e776f3f66dccf41616165d5e03634758a3a061ec4f6eeb0dae0ff152" gracePeriod=45 Nov 03 21:09:50.417654 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:50.406585 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-499187-55f6b9c44c-mmhjm" podUID="ad2a35f4-15e4-4130-ab26-9baba3182241" containerName="payload-pre-processing" containerID="cri-o://aa5fc156bc5935b6331858f52c7586b02321bd59396a6012a3e88dbb666effdc" gracePeriod=25 Nov 03 21:09:50.436571 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:50.434660 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" Nov 03 21:09:50.441467 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:50.441357 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" Nov 03 21:09:50.525301 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:50.525212 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ee1f028-8503-4034-9954-eb06fcf402e7-home\" (UniqueName: \"kubernetes.io/empty-dir/8ee1f028-8503-4034-9954-eb06fcf402e7-home\") pod \"8ee1f028-8503-4034-9954-eb06fcf402e7\" (UID: \"8ee1f028-8503-4034-9954-eb06fcf402e7\") " Nov 03 21:09:50.525301 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:50.525259 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/793b8ca3-3a2e-49a1-9be4-4971ba59df09-kube-api-access-vkb5c\" (UniqueName: \"kubernetes.io/projected/793b8ca3-3a2e-49a1-9be4-4971ba59df09-kube-api-access-vkb5c\") pod \"793b8ca3-3a2e-49a1-9be4-4971ba59df09\" (UID: \"793b8ca3-3a2e-49a1-9be4-4971ba59df09\") " Nov 03 21:09:50.525551 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:50.525341 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/793b8ca3-3a2e-49a1-9be4-4971ba59df09-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/793b8ca3-3a2e-49a1-9be4-4971ba59df09-model-cache\") pod \"793b8ca3-3a2e-49a1-9be4-4971ba59df09\" (UID: \"793b8ca3-3a2e-49a1-9be4-4971ba59df09\") " Nov 03 21:09:50.525551 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:50.525366 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ee1f028-8503-4034-9954-eb06fcf402e7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ee1f028-8503-4034-9954-eb06fcf402e7-dshm\") pod \"8ee1f028-8503-4034-9954-eb06fcf402e7\" (UID: \"8ee1f028-8503-4034-9954-eb06fcf402e7\") " Nov 03 21:09:50.525551 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:50.525397 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ee1f028-8503-4034-9954-eb06fcf402e7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ee1f028-8503-4034-9954-eb06fcf402e7-tmp-dir\") pod \"8ee1f028-8503-4034-9954-eb06fcf402e7\" (UID: \"8ee1f028-8503-4034-9954-eb06fcf402e7\") " Nov 03 21:09:50.525551 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:50.525502 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ee1f028-8503-4034-9954-eb06fcf402e7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ee1f028-8503-4034-9954-eb06fcf402e7-model-cache\") pod \"8ee1f028-8503-4034-9954-eb06fcf402e7\" (UID: \"8ee1f028-8503-4034-9954-eb06fcf402e7\") " Nov 03 21:09:50.525767 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:50.525570 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/793b8ca3-3a2e-49a1-9be4-4971ba59df09-dshm\" (UniqueName: \"kubernetes.io/empty-dir/793b8ca3-3a2e-49a1-9be4-4971ba59df09-dshm\") pod \"793b8ca3-3a2e-49a1-9be4-4971ba59df09\" (UID: \"793b8ca3-3a2e-49a1-9be4-4971ba59df09\") " Nov 03 21:09:50.525767 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:50.525570 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ee1f028-8503-4034-9954-eb06fcf402e7-home" pod "8ee1f028-8503-4034-9954-eb06fcf402e7" (UID: "8ee1f028-8503-4034-9954-eb06fcf402e7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:50.525767 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:50.525645 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/793b8ca3-3a2e-49a1-9be4-4971ba59df09-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/793b8ca3-3a2e-49a1-9be4-4971ba59df09-tmp-dir\") pod \"793b8ca3-3a2e-49a1-9be4-4971ba59df09\" (UID: \"793b8ca3-3a2e-49a1-9be4-4971ba59df09\") " Nov 03 21:09:50.525767 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:50.525651 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/793b8ca3-3a2e-49a1-9be4-4971ba59df09-model-cache" pod "793b8ca3-3a2e-49a1-9be4-4971ba59df09" (UID: "793b8ca3-3a2e-49a1-9be4-4971ba59df09"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:50.525767 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:50.525681 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ee1f028-8503-4034-9954-eb06fcf402e7-kube-api-access-g5hl5\" (UniqueName: \"kubernetes.io/projected/8ee1f028-8503-4034-9954-eb06fcf402e7-kube-api-access-g5hl5\") pod \"8ee1f028-8503-4034-9954-eb06fcf402e7\" (UID: \"8ee1f028-8503-4034-9954-eb06fcf402e7\") " Nov 03 21:09:50.526056 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:50.525780 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ee1f028-8503-4034-9954-eb06fcf402e7-tls-certs\" (UniqueName: \"kubernetes.io/secret/8ee1f028-8503-4034-9954-eb06fcf402e7-tls-certs\") pod \"8ee1f028-8503-4034-9954-eb06fcf402e7\" (UID: \"8ee1f028-8503-4034-9954-eb06fcf402e7\") " Nov 03 21:09:50.526056 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:50.525786 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ee1f028-8503-4034-9954-eb06fcf402e7-tmp-dir" pod "8ee1f028-8503-4034-9954-eb06fcf402e7" (UID: "8ee1f028-8503-4034-9954-eb06fcf402e7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:50.526056 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:50.525812 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ee1f028-8503-4034-9954-eb06fcf402e7-model-cache" pod "8ee1f028-8503-4034-9954-eb06fcf402e7" (UID: "8ee1f028-8503-4034-9954-eb06fcf402e7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:50.526056 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:50.525854 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/793b8ca3-3a2e-49a1-9be4-4971ba59df09-home\" (UniqueName: \"kubernetes.io/empty-dir/793b8ca3-3a2e-49a1-9be4-4971ba59df09-home\") pod \"793b8ca3-3a2e-49a1-9be4-4971ba59df09\" (UID: \"793b8ca3-3a2e-49a1-9be4-4971ba59df09\") " Nov 03 21:09:50.526056 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:50.525861 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/793b8ca3-3a2e-49a1-9be4-4971ba59df09-tmp-dir" pod "793b8ca3-3a2e-49a1-9be4-4971ba59df09" (UID: "793b8ca3-3a2e-49a1-9be4-4971ba59df09"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:50.526056 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:50.525895 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/793b8ca3-3a2e-49a1-9be4-4971ba59df09-tls-certs\" (UniqueName: \"kubernetes.io/secret/793b8ca3-3a2e-49a1-9be4-4971ba59df09-tls-certs\") pod \"793b8ca3-3a2e-49a1-9be4-4971ba59df09\" (UID: \"793b8ca3-3a2e-49a1-9be4-4971ba59df09\") " Nov 03 21:09:50.526659 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:50.526127 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/793b8ca3-3a2e-49a1-9be4-4971ba59df09-home" pod "793b8ca3-3a2e-49a1-9be4-4971ba59df09" (UID: "793b8ca3-3a2e-49a1-9be4-4971ba59df09"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:50.526820 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:50.526797 4305 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/793b8ca3-3a2e-49a1-9be4-4971ba59df09-tmp-dir\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:50.526953 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:50.526828 4305 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/793b8ca3-3a2e-49a1-9be4-4971ba59df09-home\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:50.526953 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:50.526843 4305 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ee1f028-8503-4034-9954-eb06fcf402e7-home\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:50.526953 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:50.526855 4305 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/793b8ca3-3a2e-49a1-9be4-4971ba59df09-model-cache\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:50.526953 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:50.526869 4305 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ee1f028-8503-4034-9954-eb06fcf402e7-tmp-dir\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:50.526953 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:50.526884 4305 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ee1f028-8503-4034-9954-eb06fcf402e7-model-cache\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:50.528599 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:50.528489 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/793b8ca3-3a2e-49a1-9be4-4971ba59df09-dshm" pod "793b8ca3-3a2e-49a1-9be4-4971ba59df09" (UID: "793b8ca3-3a2e-49a1-9be4-4971ba59df09"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:50.528716 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:50.528620 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ee1f028-8503-4034-9954-eb06fcf402e7-dshm" pod "8ee1f028-8503-4034-9954-eb06fcf402e7" (UID: "8ee1f028-8503-4034-9954-eb06fcf402e7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:50.528776 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:50.528735 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/793b8ca3-3a2e-49a1-9be4-4971ba59df09-kube-api-access-vkb5c" pod "793b8ca3-3a2e-49a1-9be4-4971ba59df09" (UID: "793b8ca3-3a2e-49a1-9be4-4971ba59df09"). InnerVolumeSpecName "kube-api-access-vkb5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:09:50.528904 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:50.528833 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ee1f028-8503-4034-9954-eb06fcf402e7-kube-api-access-g5hl5" pod "8ee1f028-8503-4034-9954-eb06fcf402e7" (UID: "8ee1f028-8503-4034-9954-eb06fcf402e7"). InnerVolumeSpecName "kube-api-access-g5hl5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:09:50.529180 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:50.529159 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/793b8ca3-3a2e-49a1-9be4-4971ba59df09-tls-certs" pod "793b8ca3-3a2e-49a1-9be4-4971ba59df09" (UID: "793b8ca3-3a2e-49a1-9be4-4971ba59df09"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:09:50.529237 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:50.529200 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ee1f028-8503-4034-9954-eb06fcf402e7-tls-certs" pod "8ee1f028-8503-4034-9954-eb06fcf402e7" (UID: "8ee1f028-8503-4034-9954-eb06fcf402e7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:09:50.629064 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:50.629017 4305 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ee1f028-8503-4034-9954-eb06fcf402e7-dshm\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:50.629064 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:50.629061 4305 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/793b8ca3-3a2e-49a1-9be4-4971ba59df09-dshm\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:50.629349 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:50.629077 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5hl5\" (UniqueName: \"kubernetes.io/projected/8ee1f028-8503-4034-9954-eb06fcf402e7-kube-api-access-g5hl5\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:50.629349 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:50.629092 4305 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ee1f028-8503-4034-9954-eb06fcf402e7-tls-certs\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:50.629349 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:50.629107 4305 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/793b8ca3-3a2e-49a1-9be4-4971ba59df09-tls-certs\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:50.629349 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:50.629120 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkb5c\" (UniqueName: \"kubernetes.io/projected/793b8ca3-3a2e-49a1-9be4-4971ba59df09-kube-api-access-vkb5c\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:51.000860 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:51.000826 4305 generic.go:356] "Generic (PLEG): container finished" podID="8ee1f028-8503-4034-9954-eb06fcf402e7" containerID="af86062f6aee09ba31fc2e3ec73039f924e12d291f1051c73faa10d4218e16c6" exitCode=0 Nov 03 21:09:51.001359 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:51.000876 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" Nov 03 21:09:51.001359 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:51.000889 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" event={"ID":"8ee1f028-8503-4034-9954-eb06fcf402e7","Type":"ContainerDied","Data":"af86062f6aee09ba31fc2e3ec73039f924e12d291f1051c73faa10d4218e16c6"} Nov 03 21:09:51.001359 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:51.000942 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr" event={"ID":"8ee1f028-8503-4034-9954-eb06fcf402e7","Type":"ContainerDied","Data":"3623c894e94a7aeff03abc649813f12557f459ad647bcc40bc73c1d8223a0501"} Nov 03 21:09:51.001359 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:51.000968 4305 scope.go:122] "RemoveContainer" containerID="af86062f6aee09ba31fc2e3ec73039f924e12d291f1051c73faa10d4218e16c6" Nov 03 21:09:51.003046 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:51.003017 4305 generic.go:356] "Generic (PLEG): container finished" podID="793b8ca3-3a2e-49a1-9be4-4971ba59df09" containerID="2b093b23e776f3f66dccf41616165d5e03634758a3a061ec4f6eeb0dae0ff152" exitCode=0 Nov 03 21:09:51.003186 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:51.003072 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" Nov 03 21:09:51.003186 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:51.003108 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" event={"ID":"793b8ca3-3a2e-49a1-9be4-4971ba59df09","Type":"ContainerDied","Data":"2b093b23e776f3f66dccf41616165d5e03634758a3a061ec4f6eeb0dae0ff152"} Nov 03 21:09:51.003186 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:51.003165 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" event={"ID":"793b8ca3-3a2e-49a1-9be4-4971ba59df09","Type":"ContainerDied","Data":"56998c882f0462d0edc1b03d6b6819e0ccde86c0e76a4b932e98e18c1f54e96f"} Nov 03 21:09:51.026958 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:51.026935 4305 scope.go:122] "RemoveContainer" containerID="af86062f6aee09ba31fc2e3ec73039f924e12d291f1051c73faa10d4218e16c6" Nov 03 21:09:51.027311 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:09:51.027289 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af86062f6aee09ba31fc2e3ec73039f924e12d291f1051c73faa10d4218e16c6\": container with ID starting with af86062f6aee09ba31fc2e3ec73039f924e12d291f1051c73faa10d4218e16c6 not found: ID does not exist" containerID="af86062f6aee09ba31fc2e3ec73039f924e12d291f1051c73faa10d4218e16c6" Nov 03 21:09:51.027373 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:51.027324 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af86062f6aee09ba31fc2e3ec73039f924e12d291f1051c73faa10d4218e16c6"} err="failed to get container status \"af86062f6aee09ba31fc2e3ec73039f924e12d291f1051c73faa10d4218e16c6\": rpc error: code = NotFound desc = could not find container \"af86062f6aee09ba31fc2e3ec73039f924e12d291f1051c73faa10d4218e16c6\": container with ID starting with af86062f6aee09ba31fc2e3ec73039f924e12d291f1051c73faa10d4218e16c6 not found: ID does not exist" Nov 03 21:09:51.027373 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:51.027344 4305 scope.go:122] "RemoveContainer" containerID="2b093b23e776f3f66dccf41616165d5e03634758a3a061ec4f6eeb0dae0ff152" Nov 03 21:09:51.037706 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:51.037676 4305 scope.go:122] "RemoveContainer" containerID="2b093b23e776f3f66dccf41616165d5e03634758a3a061ec4f6eeb0dae0ff152" Nov 03 21:09:51.038058 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:09:51.038038 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b093b23e776f3f66dccf41616165d5e03634758a3a061ec4f6eeb0dae0ff152\": container with ID starting with 2b093b23e776f3f66dccf41616165d5e03634758a3a061ec4f6eeb0dae0ff152 not found: ID does not exist" containerID="2b093b23e776f3f66dccf41616165d5e03634758a3a061ec4f6eeb0dae0ff152" Nov 03 21:09:51.038125 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:51.038067 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b093b23e776f3f66dccf41616165d5e03634758a3a061ec4f6eeb0dae0ff152"} err="failed to get container status \"2b093b23e776f3f66dccf41616165d5e03634758a3a061ec4f6eeb0dae0ff152\": rpc error: code = NotFound desc = could not find container \"2b093b23e776f3f66dccf41616165d5e03634758a3a061ec4f6eeb0dae0ff152\": container with ID starting with 2b093b23e776f3f66dccf41616165d5e03634758a3a061ec4f6eeb0dae0ff152 not found: ID does not exist" Nov 03 21:09:51.220591 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:51.220498 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8"] Nov 03 21:09:51.230472 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:51.230422 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8"] Nov 03 21:09:51.351105 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:51.351069 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr"] Nov 03 21:09:51.358943 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:51.358894 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-499187/distinct-w0-499187-kserve-8644578469-4zqnr"] Nov 03 21:09:51.362246 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:51.362188 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-499187/simulator-w0-499187-kserve-677fb598c-cxgd8" podUID="793b8ca3-3a2e-49a1-9be4-4971ba59df09" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:09:52.896725 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:52.896689 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="793b8ca3-3a2e-49a1-9be4-4971ba59df09" path="/var/lib/kubelet/pods/793b8ca3-3a2e-49a1-9be4-4971ba59df09/volumes" Nov 03 21:09:52.897246 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:52.897230 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ee1f028-8503-4034-9954-eb06fcf402e7" path="/var/lib/kubelet/pods/8ee1f028-8503-4034-9954-eb06fcf402e7/volumes" Nov 03 21:09:53.148096 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:53.147995 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ca4642-56449c8659-vtm4r" podUID="6f951e66-9baa-4ad3-994f-fa1b2e0d4828" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:09:53.200382 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:53.200343 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-499187-55f6b9c44c-mmhjm" podUID="ad2a35f4-15e4-4130-ab26-9baba3182241" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:09:55.242910 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:55.242880 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ca4642-56449c8659-vtm4r" Nov 03 21:09:55.304587 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:55.304548 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f951e66-9baa-4ad3-994f-fa1b2e0d4828-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f951e66-9baa-4ad3-994f-fa1b2e0d4828-plugins-config-volume\") pod \"6f951e66-9baa-4ad3-994f-fa1b2e0d4828\" (UID: \"6f951e66-9baa-4ad3-994f-fa1b2e0d4828\") " Nov 03 21:09:55.304802 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:55.304678 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f951e66-9baa-4ad3-994f-fa1b2e0d4828-kube-api-access-z75gs\" (UniqueName: \"kubernetes.io/projected/6f951e66-9baa-4ad3-994f-fa1b2e0d4828-kube-api-access-z75gs\") pod \"6f951e66-9baa-4ad3-994f-fa1b2e0d4828\" (UID: \"6f951e66-9baa-4ad3-994f-fa1b2e0d4828\") " Nov 03 21:09:55.304932 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:55.304908 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f951e66-9baa-4ad3-994f-fa1b2e0d4828-plugins-config-volume" pod "6f951e66-9baa-4ad3-994f-fa1b2e0d4828" (UID: "6f951e66-9baa-4ad3-994f-fa1b2e0d4828"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:09:55.305275 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:55.305256 4305 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f951e66-9baa-4ad3-994f-fa1b2e0d4828-plugins-config-volume\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:55.306981 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:55.306960 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f951e66-9baa-4ad3-994f-fa1b2e0d4828-kube-api-access-z75gs" pod "6f951e66-9baa-4ad3-994f-fa1b2e0d4828" (UID: "6f951e66-9baa-4ad3-994f-fa1b2e0d4828"). InnerVolumeSpecName "kube-api-access-z75gs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:09:55.406369 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:55.406329 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z75gs\" (UniqueName: \"kubernetes.io/projected/6f951e66-9baa-4ad3-994f-fa1b2e0d4828-kube-api-access-z75gs\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:09:56.044047 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:56.044012 4305 generic.go:356] "Generic (PLEG): container finished" podID="6f951e66-9baa-4ad3-994f-fa1b2e0d4828" containerID="4e05564f2fc434141b76871a062356686d18f7075903c6117bc78779d72f6c64" exitCode=0 Nov 03 21:09:56.044248 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:56.044063 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ca4642-56449c8659-vtm4r" Nov 03 21:09:56.044248 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:56.044091 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ca4642-56449c8659-vtm4r" event={"ID":"6f951e66-9baa-4ad3-994f-fa1b2e0d4828","Type":"ContainerDied","Data":"4e05564f2fc434141b76871a062356686d18f7075903c6117bc78779d72f6c64"} Nov 03 21:09:56.044248 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:56.044143 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ca4642-56449c8659-vtm4r" event={"ID":"6f951e66-9baa-4ad3-994f-fa1b2e0d4828","Type":"ContainerDied","Data":"0d4f2adfcc0946dc8ef184d0bde60df4210634b8ff33c7dbb4293ae46f11e0f7"} Nov 03 21:09:56.044248 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:56.044165 4305 scope.go:122] "RemoveContainer" containerID="4e05564f2fc434141b76871a062356686d18f7075903c6117bc78779d72f6c64" Nov 03 21:09:56.056939 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:56.056915 4305 scope.go:122] "RemoveContainer" containerID="4e05564f2fc434141b76871a062356686d18f7075903c6117bc78779d72f6c64" Nov 03 21:09:56.057305 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:09:56.057283 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e05564f2fc434141b76871a062356686d18f7075903c6117bc78779d72f6c64\": container with ID starting with 4e05564f2fc434141b76871a062356686d18f7075903c6117bc78779d72f6c64 not found: ID does not exist" containerID="4e05564f2fc434141b76871a062356686d18f7075903c6117bc78779d72f6c64" Nov 03 21:09:56.057410 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:56.057314 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e05564f2fc434141b76871a062356686d18f7075903c6117bc78779d72f6c64"} err="failed to get container status \"4e05564f2fc434141b76871a062356686d18f7075903c6117bc78779d72f6c64\": rpc error: code = NotFound desc = could not find container \"4e05564f2fc434141b76871a062356686d18f7075903c6117bc78779d72f6c64\": container with ID starting with 4e05564f2fc434141b76871a062356686d18f7075903c6117bc78779d72f6c64 not found: ID does not exist" Nov 03 21:09:56.236822 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:56.236786 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ca4642-56449c8659-vtm4r"] Nov 03 21:09:56.240298 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:56.240277 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ca4642-56449c8659-vtm4r"] Nov 03 21:09:56.896810 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:09:56.896770 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f951e66-9baa-4ad3-994f-fa1b2e0d4828" path="/var/lib/kubelet/pods/6f951e66-9baa-4ad3-994f-fa1b2e0d4828/volumes" Nov 03 21:10:02.373485 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:02.373425 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-89bb58-z292l"] Nov 03 21:10:02.422841 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:02.422802 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-89bb58-z292l"] Nov 03 21:10:02.896926 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:02.896887 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8563a9a0-3bdf-43cb-96f1-3659a04f74d6" path="/var/lib/kubelet/pods/8563a9a0-3bdf-43cb-96f1-3659a04f74d6/volumes" Nov 03 21:10:03.199373 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:03.199276 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-499187-55f6b9c44c-mmhjm" podUID="ad2a35f4-15e4-4130-ab26-9baba3182241" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:10:10.046766 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:10.046675 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s"] Nov 03 21:10:10.047146 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:10.046940 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s" podUID="c7f2f4c8-63bb-4a07-8860-91a420ebc1ae" containerName="istio-proxy" containerID="cri-o://e15906680b84f49e5f809aa4e8071c32342f2e726bfcc00f659bd78fe8bfaa12" gracePeriod=30 Nov 03 21:10:10.716162 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:10.716084 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-499187-55f6b9c44c-mmhjm" Nov 03 21:10:10.836598 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:10.836556 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad2a35f4-15e4-4130-ab26-9baba3182241-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad2a35f4-15e4-4130-ab26-9baba3182241-plugins-config-volume\") pod \"ad2a35f4-15e4-4130-ab26-9baba3182241\" (UID: \"ad2a35f4-15e4-4130-ab26-9baba3182241\") " Nov 03 21:10:10.837092 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:10.837063 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad2a35f4-15e4-4130-ab26-9baba3182241-kube-api-access-5l4pg\" (UniqueName: \"kubernetes.io/projected/ad2a35f4-15e4-4130-ab26-9baba3182241-kube-api-access-5l4pg\") pod \"ad2a35f4-15e4-4130-ab26-9baba3182241\" (UID: \"ad2a35f4-15e4-4130-ab26-9baba3182241\") " Nov 03 21:10:10.837851 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:10.837828 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad2a35f4-15e4-4130-ab26-9baba3182241-plugins-config-volume" pod "ad2a35f4-15e4-4130-ab26-9baba3182241" (UID: "ad2a35f4-15e4-4130-ab26-9baba3182241"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:10:10.839974 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:10.839949 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad2a35f4-15e4-4130-ab26-9baba3182241-kube-api-access-5l4pg" pod "ad2a35f4-15e4-4130-ab26-9baba3182241" (UID: "ad2a35f4-15e4-4130-ab26-9baba3182241"). InnerVolumeSpecName "kube-api-access-5l4pg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:10:10.939458 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:10.939401 4305 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad2a35f4-15e4-4130-ab26-9baba3182241-plugins-config-volume\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:10:10.939458 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:10.939458 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l4pg\" (UniqueName: \"kubernetes.io/projected/ad2a35f4-15e4-4130-ab26-9baba3182241-kube-api-access-5l4pg\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:10:11.185498 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:11.185456 4305 generic.go:356] "Generic (PLEG): container finished" podID="ad2a35f4-15e4-4130-ab26-9baba3182241" containerID="aa5fc156bc5935b6331858f52c7586b02321bd59396a6012a3e88dbb666effdc" exitCode=0 Nov 03 21:10:11.185985 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:11.185536 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-499187-55f6b9c44c-mmhjm" Nov 03 21:10:11.185985 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:11.185535 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-499187-55f6b9c44c-mmhjm" event={"ID":"ad2a35f4-15e4-4130-ab26-9baba3182241","Type":"ContainerDied","Data":"aa5fc156bc5935b6331858f52c7586b02321bd59396a6012a3e88dbb666effdc"} Nov 03 21:10:11.185985 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:11.185594 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-499187-55f6b9c44c-mmhjm" event={"ID":"ad2a35f4-15e4-4130-ab26-9baba3182241","Type":"ContainerDied","Data":"88e3d4f4dfad34ed6806da70a6926c05ee1531e7a966fccecc116710208aadb7"} Nov 03 21:10:11.185985 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:11.185619 4305 scope.go:122] "RemoveContainer" containerID="aa5fc156bc5935b6331858f52c7586b02321bd59396a6012a3e88dbb666effdc" Nov 03 21:10:11.196426 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:11.196404 4305 scope.go:122] "RemoveContainer" containerID="aa5fc156bc5935b6331858f52c7586b02321bd59396a6012a3e88dbb666effdc" Nov 03 21:10:11.196826 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:10:11.196803 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa5fc156bc5935b6331858f52c7586b02321bd59396a6012a3e88dbb666effdc\": container with ID starting with aa5fc156bc5935b6331858f52c7586b02321bd59396a6012a3e88dbb666effdc not found: ID does not exist" containerID="aa5fc156bc5935b6331858f52c7586b02321bd59396a6012a3e88dbb666effdc" Nov 03 21:10:11.196916 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:11.196835 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa5fc156bc5935b6331858f52c7586b02321bd59396a6012a3e88dbb666effdc"} err="failed to get container status \"aa5fc156bc5935b6331858f52c7586b02321bd59396a6012a3e88dbb666effdc\": rpc error: code = NotFound desc = could not find container \"aa5fc156bc5935b6331858f52c7586b02321bd59396a6012a3e88dbb666effdc\": container with ID starting with aa5fc156bc5935b6331858f52c7586b02321bd59396a6012a3e88dbb666effdc not found: ID does not exist" Nov 03 21:10:11.354211 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:11.354170 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-499187-55f6b9c44c-mmhjm"] Nov 03 21:10:11.359786 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:11.359748 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-499187-55f6b9c44c-mmhjm"] Nov 03 21:10:12.697449 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:12.697404 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr"] Nov 03 21:10:12.698060 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:12.698044 4305 cpu_manager.go:414] "containerMap: removing container" podUID="c31b220a-debb-4e97-9dd9-03c304faf447" containerName="revoke-keys" Nov 03 21:10:12.698103 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:12.698061 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c31b220a-debb-4e97-9dd9-03c304faf447" containerName="revoke-keys" Nov 03 21:10:12.698103 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:12.698079 4305 cpu_manager.go:414] "containerMap: removing container" podUID="c6f56404-e1ad-4521-b8ba-ab22614197a0" containerName="revoke-keys" Nov 03 21:10:12.698103 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:12.698084 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6f56404-e1ad-4521-b8ba-ab22614197a0" containerName="revoke-keys" Nov 03 21:10:12.698103 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:12.698101 4305 cpu_manager.go:414] "containerMap: removing container" podUID="793b8ca3-3a2e-49a1-9be4-4971ba59df09" containerName="main" Nov 03 21:10:12.698228 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:12.698106 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="793b8ca3-3a2e-49a1-9be4-4971ba59df09" containerName="main" Nov 03 21:10:12.698228 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:12.698121 4305 cpu_manager.go:414] "containerMap: removing container" podUID="ad2a35f4-15e4-4130-ab26-9baba3182241" containerName="payload-pre-processing" Nov 03 21:10:12.698228 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:12.698127 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad2a35f4-15e4-4130-ab26-9baba3182241" containerName="payload-pre-processing" Nov 03 21:10:12.698228 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:12.698142 4305 cpu_manager.go:414] "containerMap: removing container" podUID="6f951e66-9baa-4ad3-994f-fa1b2e0d4828" containerName="payload-pre-processing" Nov 03 21:10:12.698228 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:12.698149 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f951e66-9baa-4ad3-994f-fa1b2e0d4828" containerName="payload-pre-processing" Nov 03 21:10:12.698228 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:12.698159 4305 cpu_manager.go:414] "containerMap: removing container" podUID="8ee1f028-8503-4034-9954-eb06fcf402e7" containerName="main" Nov 03 21:10:12.698228 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:12.698163 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ee1f028-8503-4034-9954-eb06fcf402e7" containerName="main" Nov 03 21:10:12.698479 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:12.698244 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad2a35f4-15e4-4130-ab26-9baba3182241" containerName="payload-pre-processing" Nov 03 21:10:12.698479 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:12.698254 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f951e66-9baa-4ad3-994f-fa1b2e0d4828" containerName="payload-pre-processing" Nov 03 21:10:12.698479 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:12.698260 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ee1f028-8503-4034-9954-eb06fcf402e7" containerName="main" Nov 03 21:10:12.698479 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:12.698269 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="c31b220a-debb-4e97-9dd9-03c304faf447" containerName="revoke-keys" Nov 03 21:10:12.698479 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:12.698277 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6f56404-e1ad-4521-b8ba-ab22614197a0" containerName="revoke-keys" Nov 03 21:10:12.698479 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:12.698285 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="793b8ca3-3a2e-49a1-9be4-4971ba59df09" containerName="main" Nov 03 21:10:12.703204 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:12.703181 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr" Nov 03 21:10:12.760344 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:12.760313 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-32c41e-openshift-default-dockercfg-ddnj8\"" Nov 03 21:10:12.804198 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:12.804148 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/953874ed-6376-4775-b514-b09ecb325824-credential-socket\") pod \"e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr\" (UID: \"953874ed-6376-4775-b514-b09ecb325824\") " pod="openshift-ingress/e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr" Nov 03 21:10:12.804410 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:12.804209 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/953874ed-6376-4775-b514-b09ecb325824-istio-data\") pod \"e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr\" (UID: \"953874ed-6376-4775-b514-b09ecb325824\") " pod="openshift-ingress/e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr" Nov 03 21:10:12.804410 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:12.804335 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/953874ed-6376-4775-b514-b09ecb325824-istio-envoy\") pod \"e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr\" (UID: \"953874ed-6376-4775-b514-b09ecb325824\") " pod="openshift-ingress/e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr" Nov 03 21:10:12.804410 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:12.804406 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/953874ed-6376-4775-b514-b09ecb325824-workload-certs\") pod \"e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr\" (UID: \"953874ed-6376-4775-b514-b09ecb325824\") " pod="openshift-ingress/e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr" Nov 03 21:10:12.804649 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:12.804627 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/953874ed-6376-4775-b514-b09ecb325824-istio-token\") pod \"e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr\" (UID: \"953874ed-6376-4775-b514-b09ecb325824\") " pod="openshift-ingress/e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr" Nov 03 21:10:12.804710 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:12.804688 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/953874ed-6376-4775-b514-b09ecb325824-workload-socket\") pod \"e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr\" (UID: \"953874ed-6376-4775-b514-b09ecb325824\") " pod="openshift-ingress/e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr" Nov 03 21:10:12.804777 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:12.804731 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/953874ed-6376-4775-b514-b09ecb325824-istio-podinfo\") pod \"e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr\" (UID: \"953874ed-6376-4775-b514-b09ecb325824\") " pod="openshift-ingress/e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr" Nov 03 21:10:12.804834 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:12.804819 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjqvc\" (UniqueName: \"kubernetes.io/projected/953874ed-6376-4775-b514-b09ecb325824-kube-api-access-sjqvc\") pod \"e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr\" (UID: \"953874ed-6376-4775-b514-b09ecb325824\") " pod="openshift-ingress/e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr" Nov 03 21:10:12.804895 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:12.804880 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/953874ed-6376-4775-b514-b09ecb325824-istiod-ca-cert\") pod \"e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr\" (UID: \"953874ed-6376-4775-b514-b09ecb325824\") " pod="openshift-ingress/e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr" Nov 03 21:10:12.813134 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:12.813100 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr"] Nov 03 21:10:12.900350 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:12.900309 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad2a35f4-15e4-4130-ab26-9baba3182241" path="/var/lib/kubelet/pods/ad2a35f4-15e4-4130-ab26-9baba3182241/volumes" Nov 03 21:10:12.906472 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:12.906405 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/953874ed-6376-4775-b514-b09ecb325824-istio-token\") pod \"e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr\" (UID: \"953874ed-6376-4775-b514-b09ecb325824\") " pod="openshift-ingress/e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr" Nov 03 21:10:12.906664 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:12.906498 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/953874ed-6376-4775-b514-b09ecb325824-workload-socket\") pod \"e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr\" (UID: \"953874ed-6376-4775-b514-b09ecb325824\") " pod="openshift-ingress/e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr" Nov 03 21:10:12.906664 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:12.906529 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/953874ed-6376-4775-b514-b09ecb325824-istio-podinfo\") pod \"e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr\" (UID: \"953874ed-6376-4775-b514-b09ecb325824\") " pod="openshift-ingress/e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr" Nov 03 21:10:12.906664 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:12.906573 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjqvc\" (UniqueName: \"kubernetes.io/projected/953874ed-6376-4775-b514-b09ecb325824-kube-api-access-sjqvc\") pod \"e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr\" (UID: \"953874ed-6376-4775-b514-b09ecb325824\") " pod="openshift-ingress/e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr" Nov 03 21:10:12.906664 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:12.906634 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/953874ed-6376-4775-b514-b09ecb325824-istiod-ca-cert\") pod \"e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr\" (UID: \"953874ed-6376-4775-b514-b09ecb325824\") " pod="openshift-ingress/e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr" Nov 03 21:10:12.906918 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:12.906709 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/953874ed-6376-4775-b514-b09ecb325824-credential-socket\") pod \"e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr\" (UID: \"953874ed-6376-4775-b514-b09ecb325824\") " pod="openshift-ingress/e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr" Nov 03 21:10:12.906918 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:12.906736 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/953874ed-6376-4775-b514-b09ecb325824-istio-data\") pod \"e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr\" (UID: \"953874ed-6376-4775-b514-b09ecb325824\") " pod="openshift-ingress/e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr" Nov 03 21:10:12.906918 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:12.906808 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/953874ed-6376-4775-b514-b09ecb325824-istio-envoy\") pod \"e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr\" (UID: \"953874ed-6376-4775-b514-b09ecb325824\") " pod="openshift-ingress/e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr" Nov 03 21:10:12.906918 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:12.906894 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/953874ed-6376-4775-b514-b09ecb325824-workload-certs\") pod \"e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr\" (UID: \"953874ed-6376-4775-b514-b09ecb325824\") " pod="openshift-ingress/e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr" Nov 03 21:10:12.907409 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:12.907380 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/953874ed-6376-4775-b514-b09ecb325824-workload-socket\") pod \"e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr\" (UID: \"953874ed-6376-4775-b514-b09ecb325824\") " pod="openshift-ingress/e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr" Nov 03 21:10:12.907588 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:12.907390 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/953874ed-6376-4775-b514-b09ecb325824-workload-certs\") pod \"e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr\" (UID: \"953874ed-6376-4775-b514-b09ecb325824\") " pod="openshift-ingress/e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr" Nov 03 21:10:12.908061 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:12.908036 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/953874ed-6376-4775-b514-b09ecb325824-istiod-ca-cert\") pod \"e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr\" (UID: \"953874ed-6376-4775-b514-b09ecb325824\") " pod="openshift-ingress/e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr" Nov 03 21:10:12.908316 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:12.908290 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/953874ed-6376-4775-b514-b09ecb325824-istio-data\") pod \"e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr\" (UID: \"953874ed-6376-4775-b514-b09ecb325824\") " pod="openshift-ingress/e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr" Nov 03 21:10:12.908536 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:12.908515 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/953874ed-6376-4775-b514-b09ecb325824-credential-socket\") pod \"e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr\" (UID: \"953874ed-6376-4775-b514-b09ecb325824\") " pod="openshift-ingress/e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr" Nov 03 21:10:12.910592 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:12.910567 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/953874ed-6376-4775-b514-b09ecb325824-istio-envoy\") pod \"e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr\" (UID: \"953874ed-6376-4775-b514-b09ecb325824\") " pod="openshift-ingress/e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr" Nov 03 21:10:12.910717 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:12.910690 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/953874ed-6376-4775-b514-b09ecb325824-istio-podinfo\") pod \"e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr\" (UID: \"953874ed-6376-4775-b514-b09ecb325824\") " pod="openshift-ingress/e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr" Nov 03 21:10:12.969039 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:12.968964 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/953874ed-6376-4775-b514-b09ecb325824-istio-token\") pod \"e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr\" (UID: \"953874ed-6376-4775-b514-b09ecb325824\") " pod="openshift-ingress/e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr" Nov 03 21:10:12.969188 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:12.969117 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjqvc\" (UniqueName: \"kubernetes.io/projected/953874ed-6376-4775-b514-b09ecb325824-kube-api-access-sjqvc\") pod \"e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr\" (UID: \"953874ed-6376-4775-b514-b09ecb325824\") " pod="openshift-ingress/e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr" Nov 03 21:10:13.019158 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:13.019121 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr" Nov 03 21:10:13.474936 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:13.474904 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr"] Nov 03 21:10:13.475748 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:10:13.475717 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod953874ed_6376_4775_b514_b09ecb325824.slice/crio-77b81da472f95c78595e484469689e647bf1d955fc3c4f7c2729b0c1b62085bc WatchSource:0}: Error finding container 77b81da472f95c78595e484469689e647bf1d955fc3c4f7c2729b0c1b62085bc: Status 404 returned error can't find the container with id 77b81da472f95c78595e484469689e647bf1d955fc3c4f7c2729b0c1b62085bc Nov 03 21:10:13.478347 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:13.478314 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:10:13.478419 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:13.478383 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:10:13.478419 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:13.478414 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:10:13.653907 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:13.653869 4305 patch_prober.go:29] interesting pod/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:10:13.654069 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:13.653933 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s" podUID="c7f2f4c8-63bb-4a07-8860-91a420ebc1ae" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:10:14.210569 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:14.210529 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr" event={"ID":"953874ed-6376-4775-b514-b09ecb325824","Type":"ContainerStarted","Data":"ce43d975478cc8a99b640aee6c05a90a6cf327d4f1d6852ae96596c44910f64d"} Nov 03 21:10:14.211070 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:14.210599 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr" event={"ID":"953874ed-6376-4775-b514-b09ecb325824","Type":"ContainerStarted","Data":"77b81da472f95c78595e484469689e647bf1d955fc3c4f7c2729b0c1b62085bc"} Nov 03 21:10:14.336204 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:14.336151 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr" podStartSLOduration=2.336134894 podStartE2EDuration="2.336134894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:10:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:10:14.33397501 +0000 UTC m=+1849.559294800" watchObservedRunningTime="2026-11-03 21:10:14.336134894 +0000 UTC m=+1849.561454684" Nov 03 21:10:15.020343 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:15.020300 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr" Nov 03 21:10:15.026613 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:15.026584 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr" Nov 03 21:10:15.220396 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:15.220362 4305 generic.go:356] "Generic (PLEG): container finished" podID="c7f2f4c8-63bb-4a07-8860-91a420ebc1ae" containerID="e15906680b84f49e5f809aa4e8071c32342f2e726bfcc00f659bd78fe8bfaa12" exitCode=0 Nov 03 21:10:15.220863 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:15.220408 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s" event={"ID":"c7f2f4c8-63bb-4a07-8860-91a420ebc1ae","Type":"ContainerDied","Data":"e15906680b84f49e5f809aa4e8071c32342f2e726bfcc00f659bd78fe8bfaa12"} Nov 03 21:10:15.220863 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:15.220824 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr" Nov 03 21:10:15.223712 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:15.223689 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr" Nov 03 21:10:15.322205 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:15.322175 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s" Nov 03 21:10:15.439482 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:15.439448 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7f2f4c8-63bb-4a07-8860-91a420ebc1ae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7f2f4c8-63bb-4a07-8860-91a420ebc1ae-istio-data\") pod \"c7f2f4c8-63bb-4a07-8860-91a420ebc1ae\" (UID: \"c7f2f4c8-63bb-4a07-8860-91a420ebc1ae\") " Nov 03 21:10:15.439667 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:15.439496 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7f2f4c8-63bb-4a07-8860-91a420ebc1ae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7f2f4c8-63bb-4a07-8860-91a420ebc1ae-workload-socket\") pod \"c7f2f4c8-63bb-4a07-8860-91a420ebc1ae\" (UID: \"c7f2f4c8-63bb-4a07-8860-91a420ebc1ae\") " Nov 03 21:10:15.439667 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:15.439581 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7f2f4c8-63bb-4a07-8860-91a420ebc1ae-kube-api-access-b7jsx\" (UniqueName: \"kubernetes.io/projected/c7f2f4c8-63bb-4a07-8860-91a420ebc1ae-kube-api-access-b7jsx\") pod \"c7f2f4c8-63bb-4a07-8860-91a420ebc1ae\" (UID: \"c7f2f4c8-63bb-4a07-8860-91a420ebc1ae\") " Nov 03 21:10:15.439667 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:15.439607 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7f2f4c8-63bb-4a07-8860-91a420ebc1ae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7f2f4c8-63bb-4a07-8860-91a420ebc1ae-istiod-ca-cert\") pod \"c7f2f4c8-63bb-4a07-8860-91a420ebc1ae\" (UID: \"c7f2f4c8-63bb-4a07-8860-91a420ebc1ae\") " Nov 03 21:10:15.439667 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:15.439650 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7f2f4c8-63bb-4a07-8860-91a420ebc1ae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7f2f4c8-63bb-4a07-8860-91a420ebc1ae-istio-envoy\") pod \"c7f2f4c8-63bb-4a07-8860-91a420ebc1ae\" (UID: \"c7f2f4c8-63bb-4a07-8860-91a420ebc1ae\") " Nov 03 21:10:15.439901 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:15.439712 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7f2f4c8-63bb-4a07-8860-91a420ebc1ae-istio-data" pod "c7f2f4c8-63bb-4a07-8860-91a420ebc1ae" (UID: "c7f2f4c8-63bb-4a07-8860-91a420ebc1ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:15.439901 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:15.439801 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7f2f4c8-63bb-4a07-8860-91a420ebc1ae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7f2f4c8-63bb-4a07-8860-91a420ebc1ae-workload-certs\") pod \"c7f2f4c8-63bb-4a07-8860-91a420ebc1ae\" (UID: \"c7f2f4c8-63bb-4a07-8860-91a420ebc1ae\") " Nov 03 21:10:15.439901 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:15.439831 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7f2f4c8-63bb-4a07-8860-91a420ebc1ae-workload-socket" pod "c7f2f4c8-63bb-4a07-8860-91a420ebc1ae" (UID: "c7f2f4c8-63bb-4a07-8860-91a420ebc1ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:15.439901 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:15.439893 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7f2f4c8-63bb-4a07-8860-91a420ebc1ae-istio-token\" (UniqueName: \"kubernetes.io/projected/c7f2f4c8-63bb-4a07-8860-91a420ebc1ae-istio-token\") pod \"c7f2f4c8-63bb-4a07-8860-91a420ebc1ae\" (UID: \"c7f2f4c8-63bb-4a07-8860-91a420ebc1ae\") " Nov 03 21:10:15.440113 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:15.440059 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7f2f4c8-63bb-4a07-8860-91a420ebc1ae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7f2f4c8-63bb-4a07-8860-91a420ebc1ae-credential-socket\") pod \"c7f2f4c8-63bb-4a07-8860-91a420ebc1ae\" (UID: \"c7f2f4c8-63bb-4a07-8860-91a420ebc1ae\") " Nov 03 21:10:15.440113 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:15.440084 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7f2f4c8-63bb-4a07-8860-91a420ebc1ae-istiod-ca-cert" pod "c7f2f4c8-63bb-4a07-8860-91a420ebc1ae" (UID: "c7f2f4c8-63bb-4a07-8860-91a420ebc1ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:10:15.440113 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:15.440109 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c7f2f4c8-63bb-4a07-8860-91a420ebc1ae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7f2f4c8-63bb-4a07-8860-91a420ebc1ae-istio-podinfo\") pod \"c7f2f4c8-63bb-4a07-8860-91a420ebc1ae\" (UID: \"c7f2f4c8-63bb-4a07-8860-91a420ebc1ae\") " Nov 03 21:10:15.440337 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:15.440107 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7f2f4c8-63bb-4a07-8860-91a420ebc1ae-workload-certs" pod "c7f2f4c8-63bb-4a07-8860-91a420ebc1ae" (UID: "c7f2f4c8-63bb-4a07-8860-91a420ebc1ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:15.440403 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:15.440364 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7f2f4c8-63bb-4a07-8860-91a420ebc1ae-credential-socket" pod "c7f2f4c8-63bb-4a07-8860-91a420ebc1ae" (UID: "c7f2f4c8-63bb-4a07-8860-91a420ebc1ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:15.440807 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:15.440786 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7f2f4c8-63bb-4a07-8860-91a420ebc1ae-istio-data\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:10:15.440807 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:15.440810 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7f2f4c8-63bb-4a07-8860-91a420ebc1ae-workload-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:10:15.441035 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:15.440824 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7f2f4c8-63bb-4a07-8860-91a420ebc1ae-istiod-ca-cert\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:10:15.441035 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:15.440837 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7f2f4c8-63bb-4a07-8860-91a420ebc1ae-workload-certs\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:10:15.441035 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:15.440850 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7f2f4c8-63bb-4a07-8860-91a420ebc1ae-credential-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:10:15.442355 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:15.442330 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7f2f4c8-63bb-4a07-8860-91a420ebc1ae-istio-podinfo" pod "c7f2f4c8-63bb-4a07-8860-91a420ebc1ae" (UID: "c7f2f4c8-63bb-4a07-8860-91a420ebc1ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:10:15.442504 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:15.442462 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7f2f4c8-63bb-4a07-8860-91a420ebc1ae-istio-envoy" pod "c7f2f4c8-63bb-4a07-8860-91a420ebc1ae" (UID: "c7f2f4c8-63bb-4a07-8860-91a420ebc1ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:15.442965 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:15.442943 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7f2f4c8-63bb-4a07-8860-91a420ebc1ae-kube-api-access-b7jsx" pod "c7f2f4c8-63bb-4a07-8860-91a420ebc1ae" (UID: "c7f2f4c8-63bb-4a07-8860-91a420ebc1ae"). InnerVolumeSpecName "kube-api-access-b7jsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:10:15.443079 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:15.442973 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7f2f4c8-63bb-4a07-8860-91a420ebc1ae-istio-token" pod "c7f2f4c8-63bb-4a07-8860-91a420ebc1ae" (UID: "c7f2f4c8-63bb-4a07-8860-91a420ebc1ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:10:15.542916 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:15.542811 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7f2f4c8-63bb-4a07-8860-91a420ebc1ae-istio-envoy\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:10:15.542916 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:15.542844 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7f2f4c8-63bb-4a07-8860-91a420ebc1ae-istio-token\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:10:15.542916 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:15.542861 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7f2f4c8-63bb-4a07-8860-91a420ebc1ae-istio-podinfo\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:10:15.542916 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:15.542877 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7jsx\" (UniqueName: \"kubernetes.io/projected/c7f2f4c8-63bb-4a07-8860-91a420ebc1ae-kube-api-access-b7jsx\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:10:16.230050 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:16.230019 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s" Nov 03 21:10:16.230596 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:16.230055 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s" event={"ID":"c7f2f4c8-63bb-4a07-8860-91a420ebc1ae","Type":"ContainerDied","Data":"6f2ab1445494e6840b5d702a77523203ef049cb8655d5606145aa198b638d09f"} Nov 03 21:10:16.230596 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:16.230134 4305 scope.go:122] "RemoveContainer" containerID="e15906680b84f49e5f809aa4e8071c32342f2e726bfcc00f659bd78fe8bfaa12" Nov 03 21:10:16.323630 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:10:16.323587 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:10:16.395745 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:16.395709 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s"] Nov 03 21:10:16.447020 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:16.446982 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-89bb58-openshift-default-7f67949dc5-qtc7s"] Nov 03 21:10:16.897085 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:16.897050 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7f2f4c8-63bb-4a07-8860-91a420ebc1ae" path="/var/lib/kubelet/pods/c7f2f4c8-63bb-4a07-8860-91a420ebc1ae/volumes" Nov 03 21:10:22.684270 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:22.684226 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs"] Nov 03 21:10:22.685053 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:22.685030 4305 cpu_manager.go:414] "containerMap: removing container" podUID="c7f2f4c8-63bb-4a07-8860-91a420ebc1ae" containerName="istio-proxy" Nov 03 21:10:22.685053 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:22.685054 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7f2f4c8-63bb-4a07-8860-91a420ebc1ae" containerName="istio-proxy" Nov 03 21:10:22.685228 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:22.685176 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7f2f4c8-63bb-4a07-8860-91a420ebc1ae" containerName="istio-proxy" Nov 03 21:10:22.690192 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:22.690153 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs" Nov 03 21:10:22.834461 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:22.833369 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-1e5494-openshift-default-dockercfg-9sqlg\"" Nov 03 21:10:22.834667 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:22.834600 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4063115-7846-42ea-8bea-4684c27b132a-credential-socket\") pod \"e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs\" (UID: \"e4063115-7846-42ea-8bea-4684c27b132a\") " pod="openshift-ingress/e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs" Nov 03 21:10:22.834729 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:22.834716 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4063115-7846-42ea-8bea-4684c27b132a-istio-token\") pod \"e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs\" (UID: \"e4063115-7846-42ea-8bea-4684c27b132a\") " pod="openshift-ingress/e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs" Nov 03 21:10:22.834784 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:22.834749 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4063115-7846-42ea-8bea-4684c27b132a-workload-socket\") pod \"e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs\" (UID: \"e4063115-7846-42ea-8bea-4684c27b132a\") " pod="openshift-ingress/e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs" Nov 03 21:10:22.834784 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:22.834776 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4063115-7846-42ea-8bea-4684c27b132a-istio-envoy\") pod \"e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs\" (UID: \"e4063115-7846-42ea-8bea-4684c27b132a\") " pod="openshift-ingress/e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs" Nov 03 21:10:22.834884 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:22.834825 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4063115-7846-42ea-8bea-4684c27b132a-workload-certs\") pod \"e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs\" (UID: \"e4063115-7846-42ea-8bea-4684c27b132a\") " pod="openshift-ingress/e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs" Nov 03 21:10:22.841505 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:22.834993 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhr9r\" (UniqueName: \"kubernetes.io/projected/e4063115-7846-42ea-8bea-4684c27b132a-kube-api-access-qhr9r\") pod \"e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs\" (UID: \"e4063115-7846-42ea-8bea-4684c27b132a\") " pod="openshift-ingress/e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs" Nov 03 21:10:22.841505 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:22.835041 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4063115-7846-42ea-8bea-4684c27b132a-istiod-ca-cert\") pod \"e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs\" (UID: \"e4063115-7846-42ea-8bea-4684c27b132a\") " pod="openshift-ingress/e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs" Nov 03 21:10:22.841505 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:22.835081 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4063115-7846-42ea-8bea-4684c27b132a-istio-podinfo\") pod \"e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs\" (UID: \"e4063115-7846-42ea-8bea-4684c27b132a\") " pod="openshift-ingress/e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs" Nov 03 21:10:22.841505 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:22.835478 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4063115-7846-42ea-8bea-4684c27b132a-istio-data\") pod \"e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs\" (UID: \"e4063115-7846-42ea-8bea-4684c27b132a\") " pod="openshift-ingress/e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs" Nov 03 21:10:22.843025 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:22.842997 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs"] Nov 03 21:10:22.939424 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:22.939322 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4063115-7846-42ea-8bea-4684c27b132a-istio-data\") pod \"e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs\" (UID: \"e4063115-7846-42ea-8bea-4684c27b132a\") " pod="openshift-ingress/e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs" Nov 03 21:10:22.939424 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:22.939370 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4063115-7846-42ea-8bea-4684c27b132a-credential-socket\") pod \"e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs\" (UID: \"e4063115-7846-42ea-8bea-4684c27b132a\") " pod="openshift-ingress/e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs" Nov 03 21:10:22.939708 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:22.939469 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4063115-7846-42ea-8bea-4684c27b132a-istio-token\") pod \"e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs\" (UID: \"e4063115-7846-42ea-8bea-4684c27b132a\") " pod="openshift-ingress/e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs" Nov 03 21:10:22.939708 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:22.939499 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4063115-7846-42ea-8bea-4684c27b132a-workload-socket\") pod \"e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs\" (UID: \"e4063115-7846-42ea-8bea-4684c27b132a\") " pod="openshift-ingress/e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs" Nov 03 21:10:22.939849 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:22.939783 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4063115-7846-42ea-8bea-4684c27b132a-istio-data\") pod \"e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs\" (UID: \"e4063115-7846-42ea-8bea-4684c27b132a\") " pod="openshift-ingress/e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs" Nov 03 21:10:22.939849 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:22.939822 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4063115-7846-42ea-8bea-4684c27b132a-istio-envoy\") pod \"e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs\" (UID: \"e4063115-7846-42ea-8bea-4684c27b132a\") " pod="openshift-ingress/e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs" Nov 03 21:10:22.939979 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:22.939858 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4063115-7846-42ea-8bea-4684c27b132a-workload-socket\") pod \"e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs\" (UID: \"e4063115-7846-42ea-8bea-4684c27b132a\") " pod="openshift-ingress/e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs" Nov 03 21:10:22.939979 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:22.939879 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4063115-7846-42ea-8bea-4684c27b132a-workload-certs\") pod \"e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs\" (UID: \"e4063115-7846-42ea-8bea-4684c27b132a\") " pod="openshift-ingress/e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs" Nov 03 21:10:22.940096 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:22.939974 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4063115-7846-42ea-8bea-4684c27b132a-credential-socket\") pod \"e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs\" (UID: \"e4063115-7846-42ea-8bea-4684c27b132a\") " pod="openshift-ingress/e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs" Nov 03 21:10:22.940188 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:22.939976 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhr9r\" (UniqueName: \"kubernetes.io/projected/e4063115-7846-42ea-8bea-4684c27b132a-kube-api-access-qhr9r\") pod \"e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs\" (UID: \"e4063115-7846-42ea-8bea-4684c27b132a\") " pod="openshift-ingress/e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs" Nov 03 21:10:22.940262 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:22.940216 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4063115-7846-42ea-8bea-4684c27b132a-istiod-ca-cert\") pod \"e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs\" (UID: \"e4063115-7846-42ea-8bea-4684c27b132a\") " pod="openshift-ingress/e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs" Nov 03 21:10:22.940262 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:22.940240 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4063115-7846-42ea-8bea-4684c27b132a-workload-certs\") pod \"e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs\" (UID: \"e4063115-7846-42ea-8bea-4684c27b132a\") " pod="openshift-ingress/e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs" Nov 03 21:10:22.940367 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:22.940261 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4063115-7846-42ea-8bea-4684c27b132a-istio-podinfo\") pod \"e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs\" (UID: \"e4063115-7846-42ea-8bea-4684c27b132a\") " pod="openshift-ingress/e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs" Nov 03 21:10:22.940849 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:22.940826 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4063115-7846-42ea-8bea-4684c27b132a-istiod-ca-cert\") pod \"e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs\" (UID: \"e4063115-7846-42ea-8bea-4684c27b132a\") " pod="openshift-ingress/e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs" Nov 03 21:10:22.942400 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:22.942374 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4063115-7846-42ea-8bea-4684c27b132a-istio-envoy\") pod \"e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs\" (UID: \"e4063115-7846-42ea-8bea-4684c27b132a\") " pod="openshift-ingress/e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs" Nov 03 21:10:22.942831 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:22.942811 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4063115-7846-42ea-8bea-4684c27b132a-istio-podinfo\") pod \"e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs\" (UID: \"e4063115-7846-42ea-8bea-4684c27b132a\") " pod="openshift-ingress/e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs" Nov 03 21:10:23.043461 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:23.029636 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4063115-7846-42ea-8bea-4684c27b132a-istio-token\") pod \"e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs\" (UID: \"e4063115-7846-42ea-8bea-4684c27b132a\") " pod="openshift-ingress/e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs" Nov 03 21:10:23.058390 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:23.058343 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhr9r\" (UniqueName: \"kubernetes.io/projected/e4063115-7846-42ea-8bea-4684c27b132a-kube-api-access-qhr9r\") pod \"e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs\" (UID: \"e4063115-7846-42ea-8bea-4684c27b132a\") " pod="openshift-ingress/e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs" Nov 03 21:10:23.141049 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:23.140709 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs" Nov 03 21:10:23.280154 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:23.280117 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:10:23.609623 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:23.609589 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs"] Nov 03 21:10:23.610366 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:10:23.610334 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4063115_7846_42ea_8bea_4684c27b132a.slice/crio-de44b3e25aa41366c6d35d7b8f91d8dc03602daf7b0eb1a23895a8a87d16d8e5 WatchSource:0}: Error finding container de44b3e25aa41366c6d35d7b8f91d8dc03602daf7b0eb1a23895a8a87d16d8e5: Status 404 returned error can't find the container with id de44b3e25aa41366c6d35d7b8f91d8dc03602daf7b0eb1a23895a8a87d16d8e5 Nov 03 21:10:23.613161 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:23.613122 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:10:23.613283 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:23.613196 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:10:23.613283 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:23.613227 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:10:23.779363 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:23.779326 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b11be461-gsj8r"] Nov 03 21:10:23.785466 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:23.785421 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11be461-gsj8r" Nov 03 21:10:23.831920 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:23.831873 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-043ccaa3-j4fhl"] Nov 03 21:10:23.837332 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:23.837298 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77vtd\"" Nov 03 21:10:23.837628 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:23.837415 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-043ccaa3-j4fhl" Nov 03 21:10:23.845824 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:23.845778 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b11be461-gsj8r"] Nov 03 21:10:23.857790 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:23.857719 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:10:23.863262 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:23.863230 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49q4q\" (UniqueName: \"kubernetes.io/projected/ae494fad-b280-451c-8669-2340a689bec8-kube-api-access-49q4q\") pod \"maas-api-revoke-sub-b11be461-gsj8r\" (UID: \"ae494fad-b280-451c-8669-2340a689bec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11be461-gsj8r" Nov 03 21:10:23.863753 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:23.863659 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzdzx\" (UniqueName: \"kubernetes.io/projected/5c2d98e6-472e-40bc-80df-efded30cbb5d-kube-api-access-bzdzx\") pod \"maas-api-revoke-sub-043ccaa3-j4fhl\" (UID: \"5c2d98e6-472e-40bc-80df-efded30cbb5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-043ccaa3-j4fhl" Nov 03 21:10:23.863937 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:23.863921 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c2d98e6-472e-40bc-80df-efded30cbb5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-043ccaa3-j4fhl\" (UID: \"5c2d98e6-472e-40bc-80df-efded30cbb5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-043ccaa3-j4fhl" Nov 03 21:10:23.864246 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:23.864231 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c2d98e6-472e-40bc-80df-efded30cbb5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-043ccaa3-j4fhl\" (UID: \"5c2d98e6-472e-40bc-80df-efded30cbb5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-043ccaa3-j4fhl" Nov 03 21:10:23.864419 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:23.864405 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae494fad-b280-451c-8669-2340a689bec8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b11be461-gsj8r\" (UID: \"ae494fad-b280-451c-8669-2340a689bec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11be461-gsj8r" Nov 03 21:10:23.864583 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:23.864569 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae494fad-b280-451c-8669-2340a689bec8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b11be461-gsj8r\" (UID: \"ae494fad-b280-451c-8669-2340a689bec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11be461-gsj8r" Nov 03 21:10:23.894979 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:23.894933 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-043ccaa3-j4fhl"] Nov 03 21:10:23.929801 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:23.929641 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:10:23.929801 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:23.929745 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:10:24.022507 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:24.002492 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c2d98e6-472e-40bc-80df-efded30cbb5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-043ccaa3-j4fhl\" (UID: \"5c2d98e6-472e-40bc-80df-efded30cbb5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-043ccaa3-j4fhl" Nov 03 21:10:24.022507 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:24.002571 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae494fad-b280-451c-8669-2340a689bec8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b11be461-gsj8r\" (UID: \"ae494fad-b280-451c-8669-2340a689bec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11be461-gsj8r" Nov 03 21:10:24.022507 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:24.002609 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae494fad-b280-451c-8669-2340a689bec8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b11be461-gsj8r\" (UID: \"ae494fad-b280-451c-8669-2340a689bec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11be461-gsj8r" Nov 03 21:10:24.022507 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:24.002792 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49q4q\" (UniqueName: \"kubernetes.io/projected/ae494fad-b280-451c-8669-2340a689bec8-kube-api-access-49q4q\") pod \"maas-api-revoke-sub-b11be461-gsj8r\" (UID: \"ae494fad-b280-451c-8669-2340a689bec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11be461-gsj8r" Nov 03 21:10:24.022507 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:24.002883 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzdzx\" (UniqueName: \"kubernetes.io/projected/5c2d98e6-472e-40bc-80df-efded30cbb5d-kube-api-access-bzdzx\") pod \"maas-api-revoke-sub-043ccaa3-j4fhl\" (UID: \"5c2d98e6-472e-40bc-80df-efded30cbb5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-043ccaa3-j4fhl" Nov 03 21:10:24.022507 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:24.002923 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c2d98e6-472e-40bc-80df-efded30cbb5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-043ccaa3-j4fhl\" (UID: \"5c2d98e6-472e-40bc-80df-efded30cbb5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-043ccaa3-j4fhl" Nov 03 21:10:24.022507 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:24.003904 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c2d98e6-472e-40bc-80df-efded30cbb5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-043ccaa3-j4fhl\" (UID: \"5c2d98e6-472e-40bc-80df-efded30cbb5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-043ccaa3-j4fhl" Nov 03 21:10:24.026455 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:24.026227 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae494fad-b280-451c-8669-2340a689bec8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b11be461-gsj8r\" (UID: \"ae494fad-b280-451c-8669-2340a689bec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11be461-gsj8r" Nov 03 21:10:24.027026 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:24.026985 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c2d98e6-472e-40bc-80df-efded30cbb5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-043ccaa3-j4fhl\" (UID: \"5c2d98e6-472e-40bc-80df-efded30cbb5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-043ccaa3-j4fhl" Nov 03 21:10:24.052918 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:24.052881 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae494fad-b280-451c-8669-2340a689bec8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b11be461-gsj8r\" (UID: \"ae494fad-b280-451c-8669-2340a689bec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11be461-gsj8r" Nov 03 21:10:24.089917 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:24.089876 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49q4q\" (UniqueName: \"kubernetes.io/projected/ae494fad-b280-451c-8669-2340a689bec8-kube-api-access-49q4q\") pod \"maas-api-revoke-sub-b11be461-gsj8r\" (UID: \"ae494fad-b280-451c-8669-2340a689bec8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11be461-gsj8r" Nov 03 21:10:24.090086 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:24.090025 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzdzx\" (UniqueName: \"kubernetes.io/projected/5c2d98e6-472e-40bc-80df-efded30cbb5d-kube-api-access-bzdzx\") pod \"maas-api-revoke-sub-043ccaa3-j4fhl\" (UID: \"5c2d98e6-472e-40bc-80df-efded30cbb5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-043ccaa3-j4fhl" Nov 03 21:10:24.107223 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:24.107184 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11be461-gsj8r" Nov 03 21:10:24.168653 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:24.168611 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-043ccaa3-j4fhl" Nov 03 21:10:24.301179 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:24.300117 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs" event={"ID":"e4063115-7846-42ea-8bea-4684c27b132a","Type":"ContainerStarted","Data":"18c8bfc24d304e2064190791989c8695d18a0eaf3e807aef35efaa067e6aa293"} Nov 03 21:10:24.301179 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:24.300219 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs" event={"ID":"e4063115-7846-42ea-8bea-4684c27b132a","Type":"ContainerStarted","Data":"de44b3e25aa41366c6d35d7b8f91d8dc03602daf7b0eb1a23895a8a87d16d8e5"} Nov 03 21:10:24.504689 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:24.504620 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs" podStartSLOduration=2.504594848 podStartE2EDuration="2.504594848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:10:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:10:24.494912392 +0000 UTC m=+1859.720232182" watchObservedRunningTime="2026-11-03 21:10:24.504594848 +0000 UTC m=+1859.729914640" Nov 03 21:10:24.543460 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:24.543401 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b11be461-gsj8r"] Nov 03 21:10:24.679644 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:10:24.679593 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c2d98e6_472e_40bc_80df_efded30cbb5d.slice/crio-68d734442d9e54c3e50b8b000027b0cf2578f424deb4ccc0b31aedb9c73c29fd WatchSource:0}: Error finding container 68d734442d9e54c3e50b8b000027b0cf2578f424deb4ccc0b31aedb9c73c29fd: Status 404 returned error can't find the container with id 68d734442d9e54c3e50b8b000027b0cf2578f424deb4ccc0b31aedb9c73c29fd Nov 03 21:10:24.680171 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:24.680139 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-043ccaa3-j4fhl"] Nov 03 21:10:25.094111 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:25.094080 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-32c41e-57d86546c-26fn2"] Nov 03 21:10:25.098570 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:25.098544 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-32c41e-57d86546c-26fn2" Nov 03 21:10:25.141284 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:25.141238 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs" Nov 03 21:10:25.153275 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:25.147774 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs" Nov 03 21:10:25.153275 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:25.151230 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-32c41e\"" Nov 03 21:10:25.153275 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:25.151322 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-32c41e-dockercfg-pkmmj\"" Nov 03 21:10:25.153275 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:25.153210 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-32c41e-57d86546c-26fn2"] Nov 03 21:10:25.244398 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:25.237171 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-32c41e-794b58b44-25fvr"] Nov 03 21:10:25.244398 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:25.241123 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8js5x\" (UniqueName: \"kubernetes.io/projected/e9c63228-f4d1-4327-9940-eed444b94174-kube-api-access-8js5x\") pod \"payload-pre-processing-e2e-shared-b-w5-32c41e-57d86546c-26fn2\" (UID: \"e9c63228-f4d1-4327-9940-eed444b94174\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-32c41e-57d86546c-26fn2" Nov 03 21:10:25.244398 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:25.241229 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9c63228-f4d1-4327-9940-eed444b94174-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-32c41e-57d86546c-26fn2\" (UID: \"e9c63228-f4d1-4327-9940-eed444b94174\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-32c41e-57d86546c-26fn2" Nov 03 21:10:25.253356 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:25.253327 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-32c41e-794b58b44-25fvr" Nov 03 21:10:25.306639 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:25.306604 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-32c41e-794b58b44-25fvr"] Nov 03 21:10:25.313666 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:25.313628 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11be461-gsj8r" event={"ID":"ae494fad-b280-451c-8669-2340a689bec8","Type":"ContainerStarted","Data":"9be7384cebb7d0b8e1a43101e103e7f8c570fa09e6e0813a0cf3f6a8e915199d"} Nov 03 21:10:25.313950 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:25.313910 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11be461-gsj8r" event={"ID":"ae494fad-b280-451c-8669-2340a689bec8","Type":"ContainerStarted","Data":"3576550af18c41228602f538522ceb52944898464d8e62cae876cae2f98a4fa9"} Nov 03 21:10:25.322302 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:25.322226 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-043ccaa3-j4fhl" event={"ID":"5c2d98e6-472e-40bc-80df-efded30cbb5d","Type":"ContainerStarted","Data":"138c464707ad1a467a5950e4214fa58eeaaa5919a4ef1e2696f57c08bcb60ccd"} Nov 03 21:10:25.322530 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:25.322514 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs" Nov 03 21:10:25.322646 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:25.322631 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-043ccaa3-j4fhl" event={"ID":"5c2d98e6-472e-40bc-80df-efded30cbb5d","Type":"ContainerStarted","Data":"68d734442d9e54c3e50b8b000027b0cf2578f424deb4ccc0b31aedb9c73c29fd"} Nov 03 21:10:25.326949 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:25.326925 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs" Nov 03 21:10:25.346376 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:25.346334 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8js5x\" (UniqueName: \"kubernetes.io/projected/e9c63228-f4d1-4327-9940-eed444b94174-kube-api-access-8js5x\") pod \"payload-pre-processing-e2e-shared-b-w5-32c41e-57d86546c-26fn2\" (UID: \"e9c63228-f4d1-4327-9940-eed444b94174\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-32c41e-57d86546c-26fn2" Nov 03 21:10:25.346736 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:25.346713 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9c63228-f4d1-4327-9940-eed444b94174-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-32c41e-57d86546c-26fn2\" (UID: \"e9c63228-f4d1-4327-9940-eed444b94174\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-32c41e-57d86546c-26fn2" Nov 03 21:10:25.348074 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:25.348048 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9c63228-f4d1-4327-9940-eed444b94174-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-32c41e-57d86546c-26fn2\" (UID: \"e9c63228-f4d1-4327-9940-eed444b94174\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-32c41e-57d86546c-26fn2" Nov 03 21:10:25.408497 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:25.408463 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8js5x\" (UniqueName: \"kubernetes.io/projected/e9c63228-f4d1-4327-9940-eed444b94174-kube-api-access-8js5x\") pod \"payload-pre-processing-e2e-shared-b-w5-32c41e-57d86546c-26fn2\" (UID: \"e9c63228-f4d1-4327-9940-eed444b94174\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-32c41e-57d86546c-26fn2" Nov 03 21:10:25.416491 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:25.416453 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-32c41e-57d86546c-26fn2" Nov 03 21:10:25.448800 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:25.448424 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/733d5cad-b828-421b-98b9-7077e03cf1d7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-32c41e-794b58b44-25fvr\" (UID: \"733d5cad-b828-421b-98b9-7077e03cf1d7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-32c41e-794b58b44-25fvr" Nov 03 21:10:25.448800 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:25.448718 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrc5r\" (UniqueName: \"kubernetes.io/projected/733d5cad-b828-421b-98b9-7077e03cf1d7-kube-api-access-hrc5r\") pod \"payload-processing-e2e-shared-b-w5-32c41e-794b58b44-25fvr\" (UID: \"733d5cad-b828-421b-98b9-7077e03cf1d7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-32c41e-794b58b44-25fvr" Nov 03 21:10:25.461907 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:25.461858 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-043ccaa3-j4fhl" podStartSLOduration=2.461838045 podStartE2EDuration="2.461838045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:10:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:10:25.45989673 +0000 UTC m=+1860.685216525" watchObservedRunningTime="2026-11-03 21:10:25.461838045 +0000 UTC m=+1860.687157835" Nov 03 21:10:25.550079 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:25.550039 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrc5r\" (UniqueName: \"kubernetes.io/projected/733d5cad-b828-421b-98b9-7077e03cf1d7-kube-api-access-hrc5r\") pod \"payload-processing-e2e-shared-b-w5-32c41e-794b58b44-25fvr\" (UID: \"733d5cad-b828-421b-98b9-7077e03cf1d7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-32c41e-794b58b44-25fvr" Nov 03 21:10:25.550268 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:25.550237 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/733d5cad-b828-421b-98b9-7077e03cf1d7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-32c41e-794b58b44-25fvr\" (UID: \"733d5cad-b828-421b-98b9-7077e03cf1d7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-32c41e-794b58b44-25fvr" Nov 03 21:10:25.550988 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:25.550959 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/733d5cad-b828-421b-98b9-7077e03cf1d7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-32c41e-794b58b44-25fvr\" (UID: \"733d5cad-b828-421b-98b9-7077e03cf1d7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-32c41e-794b58b44-25fvr" Nov 03 21:10:25.577285 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:25.577180 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11be461-gsj8r" podStartSLOduration=2.577163696 podStartE2EDuration="2.577163696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:10:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:10:25.574164301 +0000 UTC m=+1860.799484114" watchObservedRunningTime="2026-11-03 21:10:25.577163696 +0000 UTC m=+1860.802483485" Nov 03 21:10:25.610840 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:25.610799 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrc5r\" (UniqueName: \"kubernetes.io/projected/733d5cad-b828-421b-98b9-7077e03cf1d7-kube-api-access-hrc5r\") pod \"payload-processing-e2e-shared-b-w5-32c41e-794b58b44-25fvr\" (UID: \"733d5cad-b828-421b-98b9-7077e03cf1d7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-32c41e-794b58b44-25fvr" Nov 03 21:10:25.801665 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:25.801638 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-32c41e-57d86546c-26fn2"] Nov 03 21:10:25.868897 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:10:25.868806 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9c63228_f4d1_4327_9940_eed444b94174.slice/crio-ca2f5a69f78d3deb0bab0d557a486d11f3dbed4bda6af2092841d236708fdb7b WatchSource:0}: Error finding container ca2f5a69f78d3deb0bab0d557a486d11f3dbed4bda6af2092841d236708fdb7b: Status 404 returned error can't find the container with id ca2f5a69f78d3deb0bab0d557a486d11f3dbed4bda6af2092841d236708fdb7b Nov 03 21:10:25.887217 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:25.887189 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-32c41e-794b58b44-25fvr" Nov 03 21:10:26.274326 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:26.274296 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-32c41e-794b58b44-25fvr"] Nov 03 21:10:26.275713 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:10:26.275685 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod733d5cad_b828_421b_98b9_7077e03cf1d7.slice/crio-c1c29dab9d0e7ee050ec99c5447eb27e3d23a44c3d37b67cedc0376f1c70e89a WatchSource:0}: Error finding container c1c29dab9d0e7ee050ec99c5447eb27e3d23a44c3d37b67cedc0376f1c70e89a: Status 404 returned error can't find the container with id c1c29dab9d0e7ee050ec99c5447eb27e3d23a44c3d37b67cedc0376f1c70e89a Nov 03 21:10:26.329147 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:26.329112 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-32c41e-794b58b44-25fvr" event={"ID":"733d5cad-b828-421b-98b9-7077e03cf1d7","Type":"ContainerStarted","Data":"c1c29dab9d0e7ee050ec99c5447eb27e3d23a44c3d37b67cedc0376f1c70e89a"} Nov 03 21:10:26.331172 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:26.331146 4305 generic.go:356] "Generic (PLEG): container finished" podID="ae494fad-b280-451c-8669-2340a689bec8" containerID="9be7384cebb7d0b8e1a43101e103e7f8c570fa09e6e0813a0cf3f6a8e915199d" exitCode=0 Nov 03 21:10:26.331301 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:26.331280 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11be461-gsj8r" event={"ID":"ae494fad-b280-451c-8669-2340a689bec8","Type":"ContainerDied","Data":"9be7384cebb7d0b8e1a43101e103e7f8c570fa09e6e0813a0cf3f6a8e915199d"} Nov 03 21:10:26.333028 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:26.333000 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-32c41e-57d86546c-26fn2" event={"ID":"e9c63228-f4d1-4327-9940-eed444b94174","Type":"ContainerStarted","Data":"32cc2dd7c1f7e37dcc4a984af1527eadfbebde5d9f580a2356dfbcc4e58eb5be"} Nov 03 21:10:26.333195 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:26.333054 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-32c41e-57d86546c-26fn2" Nov 03 21:10:26.333195 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:26.333075 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-32c41e-57d86546c-26fn2" event={"ID":"e9c63228-f4d1-4327-9940-eed444b94174","Type":"ContainerStarted","Data":"ca2f5a69f78d3deb0bab0d557a486d11f3dbed4bda6af2092841d236708fdb7b"} Nov 03 21:10:26.334957 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:26.334930 4305 generic.go:356] "Generic (PLEG): container finished" podID="5c2d98e6-472e-40bc-80df-efded30cbb5d" containerID="138c464707ad1a467a5950e4214fa58eeaaa5919a4ef1e2696f57c08bcb60ccd" exitCode=0 Nov 03 21:10:26.335077 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:26.335003 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-043ccaa3-j4fhl" event={"ID":"5c2d98e6-472e-40bc-80df-efded30cbb5d","Type":"ContainerDied","Data":"138c464707ad1a467a5950e4214fa58eeaaa5919a4ef1e2696f57c08bcb60ccd"} Nov 03 21:10:26.689036 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:26.688969 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-32c41e-57d86546c-26fn2" podStartSLOduration=1.6889525760000001 podStartE2EDuration="1.688952576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:10:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:10:26.685614919 +0000 UTC m=+1861.910934711" watchObservedRunningTime="2026-11-03 21:10:26.688952576 +0000 UTC m=+1861.914272368" Nov 03 21:10:27.343737 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:27.343694 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-32c41e-794b58b44-25fvr" event={"ID":"733d5cad-b828-421b-98b9-7077e03cf1d7","Type":"ContainerStarted","Data":"66f77f42bbb292c2db6e29066347423f5be97d9b985c98cdbaf9b13764c5e6ea"} Nov 03 21:10:27.344256 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:27.343770 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-32c41e-794b58b44-25fvr" Nov 03 21:10:27.473535 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:27.473466 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-32c41e-794b58b44-25fvr" podStartSLOduration=2.473424531 podStartE2EDuration="2.473424531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:10:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:10:27.471072302 +0000 UTC m=+1862.696392093" watchObservedRunningTime="2026-11-03 21:10:27.473424531 +0000 UTC m=+1862.698744324" Nov 03 21:10:27.518739 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:27.518710 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-043ccaa3-j4fhl" Nov 03 21:10:27.557650 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:27.557624 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11be461-gsj8r" Nov 03 21:10:27.587066 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:27.587027 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae494fad-b280-451c-8669-2340a689bec8-kube-api-access-49q4q\" (UniqueName: \"kubernetes.io/projected/ae494fad-b280-451c-8669-2340a689bec8-kube-api-access-49q4q\") pod \"ae494fad-b280-451c-8669-2340a689bec8\" (UID: \"ae494fad-b280-451c-8669-2340a689bec8\") " Nov 03 21:10:27.587297 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:27.587258 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae494fad-b280-451c-8669-2340a689bec8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae494fad-b280-451c-8669-2340a689bec8-maas-api-xks-ca\") pod \"ae494fad-b280-451c-8669-2340a689bec8\" (UID: \"ae494fad-b280-451c-8669-2340a689bec8\") " Nov 03 21:10:27.587362 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:27.587350 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c2d98e6-472e-40bc-80df-efded30cbb5d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c2d98e6-472e-40bc-80df-efded30cbb5d-maas-api-xks-ca\") pod \"5c2d98e6-472e-40bc-80df-efded30cbb5d\" (UID: \"5c2d98e6-472e-40bc-80df-efded30cbb5d\") " Nov 03 21:10:27.587414 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:27.587399 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c2d98e6-472e-40bc-80df-efded30cbb5d-kube-api-access-bzdzx\" (UniqueName: \"kubernetes.io/projected/5c2d98e6-472e-40bc-80df-efded30cbb5d-kube-api-access-bzdzx\") pod \"5c2d98e6-472e-40bc-80df-efded30cbb5d\" (UID: \"5c2d98e6-472e-40bc-80df-efded30cbb5d\") " Nov 03 21:10:27.587979 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:27.587799 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c2d98e6-472e-40bc-80df-efded30cbb5d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c2d98e6-472e-40bc-80df-efded30cbb5d-maas-api-openshift-ca\") pod \"5c2d98e6-472e-40bc-80df-efded30cbb5d\" (UID: \"5c2d98e6-472e-40bc-80df-efded30cbb5d\") " Nov 03 21:10:27.587979 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:27.587880 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae494fad-b280-451c-8669-2340a689bec8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae494fad-b280-451c-8669-2340a689bec8-maas-api-openshift-ca\") pod \"ae494fad-b280-451c-8669-2340a689bec8\" (UID: \"ae494fad-b280-451c-8669-2340a689bec8\") " Nov 03 21:10:27.588945 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:27.588918 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae494fad-b280-451c-8669-2340a689bec8-maas-api-openshift-ca" pod "ae494fad-b280-451c-8669-2340a689bec8" (UID: "ae494fad-b280-451c-8669-2340a689bec8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:10:27.591747 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:27.591139 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c2d98e6-472e-40bc-80df-efded30cbb5d-maas-api-openshift-ca" pod "5c2d98e6-472e-40bc-80df-efded30cbb5d" (UID: "5c2d98e6-472e-40bc-80df-efded30cbb5d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:10:27.611036 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:27.610998 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae494fad-b280-451c-8669-2340a689bec8-kube-api-access-49q4q" pod "ae494fad-b280-451c-8669-2340a689bec8" (UID: "ae494fad-b280-451c-8669-2340a689bec8"). InnerVolumeSpecName "kube-api-access-49q4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:10:27.611859 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:27.611597 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c2d98e6-472e-40bc-80df-efded30cbb5d-maas-api-xks-ca" pod "5c2d98e6-472e-40bc-80df-efded30cbb5d" (UID: "5c2d98e6-472e-40bc-80df-efded30cbb5d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:10:27.611976 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:27.611868 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae494fad-b280-451c-8669-2340a689bec8-maas-api-xks-ca" pod "ae494fad-b280-451c-8669-2340a689bec8" (UID: "ae494fad-b280-451c-8669-2340a689bec8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:10:27.612291 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:27.612264 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c2d98e6-472e-40bc-80df-efded30cbb5d-kube-api-access-bzdzx" pod "5c2d98e6-472e-40bc-80df-efded30cbb5d" (UID: "5c2d98e6-472e-40bc-80df-efded30cbb5d"). InnerVolumeSpecName "kube-api-access-bzdzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:10:27.690169 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:27.690064 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c2d98e6-472e-40bc-80df-efded30cbb5d-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:10:27.690169 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:27.690098 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae494fad-b280-451c-8669-2340a689bec8-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:10:27.690169 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:27.690113 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49q4q\" (UniqueName: \"kubernetes.io/projected/ae494fad-b280-451c-8669-2340a689bec8-kube-api-access-49q4q\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:10:27.690169 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:27.690139 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae494fad-b280-451c-8669-2340a689bec8-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:10:27.690169 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:27.690347 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c2d98e6-472e-40bc-80df-efded30cbb5d-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:10:27.690169 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:27.690361 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzdzx\" (UniqueName: \"kubernetes.io/projected/5c2d98e6-472e-40bc-80df-efded30cbb5d-kube-api-access-bzdzx\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:10:28.352929 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:28.352892 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11be461-gsj8r" Nov 03 21:10:28.352929 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:28.352908 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b11be461-gsj8r" event={"ID":"ae494fad-b280-451c-8669-2340a689bec8","Type":"ContainerDied","Data":"3576550af18c41228602f538522ceb52944898464d8e62cae876cae2f98a4fa9"} Nov 03 21:10:28.353591 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:28.352990 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3576550af18c41228602f538522ceb52944898464d8e62cae876cae2f98a4fa9" Nov 03 21:10:28.354630 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:28.354612 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-043ccaa3-j4fhl" Nov 03 21:10:28.354760 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:28.354646 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-043ccaa3-j4fhl" event={"ID":"5c2d98e6-472e-40bc-80df-efded30cbb5d","Type":"ContainerDied","Data":"68d734442d9e54c3e50b8b000027b0cf2578f424deb4ccc0b31aedb9c73c29fd"} Nov 03 21:10:28.354760 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:28.354697 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68d734442d9e54c3e50b8b000027b0cf2578f424deb4ccc0b31aedb9c73c29fd" Nov 03 21:10:31.376526 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:31.376489 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-1e5494-5599ddf75f-b2h97"] Nov 03 21:10:31.377212 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:31.377192 4305 cpu_manager.go:414] "containerMap: removing container" podUID="ae494fad-b280-451c-8669-2340a689bec8" containerName="revoke-keys" Nov 03 21:10:31.377298 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:31.377216 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae494fad-b280-451c-8669-2340a689bec8" containerName="revoke-keys" Nov 03 21:10:31.377298 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:31.377243 4305 cpu_manager.go:414] "containerMap: removing container" podUID="5c2d98e6-472e-40bc-80df-efded30cbb5d" containerName="revoke-keys" Nov 03 21:10:31.377298 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:31.377249 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c2d98e6-472e-40bc-80df-efded30cbb5d" containerName="revoke-keys" Nov 03 21:10:31.377421 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:31.377373 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae494fad-b280-451c-8669-2340a689bec8" containerName="revoke-keys" Nov 03 21:10:31.377421 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:31.377387 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c2d98e6-472e-40bc-80df-efded30cbb5d" containerName="revoke-keys" Nov 03 21:10:31.381228 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:31.381207 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-1e5494-5599ddf75f-b2h97" Nov 03 21:10:31.385646 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:31.385621 4305 scope.go:122] "RemoveContainer" containerID="d788313d5dd90b3cb4e77528fe9d818d43ec2ac3c03196a26e45f9cb6b92a347" Nov 03 21:10:31.554463 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:31.443109 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-1e5494\"" Nov 03 21:10:31.554463 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:31.448616 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbnzr\" (UniqueName: \"kubernetes.io/projected/5cf1b90d-073b-4cbd-8d75-e23a6ab6590a-kube-api-access-mbnzr\") pod \"payload-pre-processing-e2e-shared-b-w6-1e5494-5599ddf75f-b2h97\" (UID: \"5cf1b90d-073b-4cbd-8d75-e23a6ab6590a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-1e5494-5599ddf75f-b2h97" Nov 03 21:10:31.554463 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:31.448747 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cf1b90d-073b-4cbd-8d75-e23a6ab6590a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-1e5494-5599ddf75f-b2h97\" (UID: \"5cf1b90d-073b-4cbd-8d75-e23a6ab6590a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-1e5494-5599ddf75f-b2h97" Nov 03 21:10:31.586533 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:31.577988 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-1e5494-dockercfg-qz2mc\"" Nov 03 21:10:31.639525 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:31.639379 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-1e5494-5599ddf75f-b2h97"] Nov 03 21:10:31.644048 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:31.642823 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cf1b90d-073b-4cbd-8d75-e23a6ab6590a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-1e5494-5599ddf75f-b2h97\" (UID: \"5cf1b90d-073b-4cbd-8d75-e23a6ab6590a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-1e5494-5599ddf75f-b2h97" Nov 03 21:10:31.644048 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:31.643174 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbnzr\" (UniqueName: \"kubernetes.io/projected/5cf1b90d-073b-4cbd-8d75-e23a6ab6590a-kube-api-access-mbnzr\") pod \"payload-pre-processing-e2e-shared-b-w6-1e5494-5599ddf75f-b2h97\" (UID: \"5cf1b90d-073b-4cbd-8d75-e23a6ab6590a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-1e5494-5599ddf75f-b2h97" Nov 03 21:10:31.644846 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:31.644807 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cf1b90d-073b-4cbd-8d75-e23a6ab6590a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-1e5494-5599ddf75f-b2h97\" (UID: \"5cf1b90d-073b-4cbd-8d75-e23a6ab6590a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-1e5494-5599ddf75f-b2h97" Nov 03 21:10:31.649542 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:31.649514 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-1e5494-7cd9cc6ffd-jqr7r"] Nov 03 21:10:31.657037 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:31.657013 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-1e5494-7cd9cc6ffd-jqr7r" Nov 03 21:10:31.658359 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:31.658334 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-1e5494-7cd9cc6ffd-jqr7r"] Nov 03 21:10:31.704700 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:31.704666 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbnzr\" (UniqueName: \"kubernetes.io/projected/5cf1b90d-073b-4cbd-8d75-e23a6ab6590a-kube-api-access-mbnzr\") pod \"payload-pre-processing-e2e-shared-b-w6-1e5494-5599ddf75f-b2h97\" (UID: \"5cf1b90d-073b-4cbd-8d75-e23a6ab6590a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-1e5494-5599ddf75f-b2h97" Nov 03 21:10:31.745026 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:31.744984 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzzqp\" (UniqueName: \"kubernetes.io/projected/dc0be43c-23aa-4e0c-b69a-34162e3fa0db-kube-api-access-hzzqp\") pod \"payload-processing-e2e-shared-b-w6-1e5494-7cd9cc6ffd-jqr7r\" (UID: \"dc0be43c-23aa-4e0c-b69a-34162e3fa0db\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-1e5494-7cd9cc6ffd-jqr7r" Nov 03 21:10:31.745198 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:31.745166 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc0be43c-23aa-4e0c-b69a-34162e3fa0db-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-1e5494-7cd9cc6ffd-jqr7r\" (UID: \"dc0be43c-23aa-4e0c-b69a-34162e3fa0db\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-1e5494-7cd9cc6ffd-jqr7r" Nov 03 21:10:31.847036 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:31.846999 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc0be43c-23aa-4e0c-b69a-34162e3fa0db-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-1e5494-7cd9cc6ffd-jqr7r\" (UID: \"dc0be43c-23aa-4e0c-b69a-34162e3fa0db\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-1e5494-7cd9cc6ffd-jqr7r" Nov 03 21:10:31.847205 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:31.847173 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzzqp\" (UniqueName: \"kubernetes.io/projected/dc0be43c-23aa-4e0c-b69a-34162e3fa0db-kube-api-access-hzzqp\") pod \"payload-processing-e2e-shared-b-w6-1e5494-7cd9cc6ffd-jqr7r\" (UID: \"dc0be43c-23aa-4e0c-b69a-34162e3fa0db\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-1e5494-7cd9cc6ffd-jqr7r" Nov 03 21:10:31.847731 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:31.847709 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc0be43c-23aa-4e0c-b69a-34162e3fa0db-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-1e5494-7cd9cc6ffd-jqr7r\" (UID: \"dc0be43c-23aa-4e0c-b69a-34162e3fa0db\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-1e5494-7cd9cc6ffd-jqr7r" Nov 03 21:10:31.908475 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:31.908422 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzzqp\" (UniqueName: \"kubernetes.io/projected/dc0be43c-23aa-4e0c-b69a-34162e3fa0db-kube-api-access-hzzqp\") pod \"payload-processing-e2e-shared-b-w6-1e5494-7cd9cc6ffd-jqr7r\" (UID: \"dc0be43c-23aa-4e0c-b69a-34162e3fa0db\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-1e5494-7cd9cc6ffd-jqr7r" Nov 03 21:10:31.978041 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:31.978000 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-1e5494-7cd9cc6ffd-jqr7r" Nov 03 21:10:31.998319 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:31.998289 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-1e5494-5599ddf75f-b2h97" Nov 03 21:10:32.369426 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:32.369334 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-1e5494-7cd9cc6ffd-jqr7r"] Nov 03 21:10:32.406213 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:10:32.406183 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc0be43c_23aa_4e0c_b69a_34162e3fa0db.slice/crio-43efc4e022c0589640d0f2363135584a2cc01592c044ca0e2af8fc75c494acdd WatchSource:0}: Error finding container 43efc4e022c0589640d0f2363135584a2cc01592c044ca0e2af8fc75c494acdd: Status 404 returned error can't find the container with id 43efc4e022c0589640d0f2363135584a2cc01592c044ca0e2af8fc75c494acdd Nov 03 21:10:32.426413 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:32.426377 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-1e5494-5599ddf75f-b2h97"] Nov 03 21:10:32.426972 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:10:32.426944 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cf1b90d_073b_4cbd_8d75_e23a6ab6590a.slice/crio-a9796911ccd9bb83300c87308df98592b5b8559d450ac2543ff79f805648e290 WatchSource:0}: Error finding container a9796911ccd9bb83300c87308df98592b5b8559d450ac2543ff79f805648e290: Status 404 returned error can't find the container with id a9796911ccd9bb83300c87308df98592b5b8559d450ac2543ff79f805648e290 Nov 03 21:10:33.396286 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:33.396248 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-1e5494-5599ddf75f-b2h97" event={"ID":"5cf1b90d-073b-4cbd-8d75-e23a6ab6590a","Type":"ContainerStarted","Data":"32e968ec56691ac3bbd2dbe3af3f96d0f9d9f7f9defa446a023eacaf634e7968"} Nov 03 21:10:33.396533 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:33.396326 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-1e5494-5599ddf75f-b2h97" event={"ID":"5cf1b90d-073b-4cbd-8d75-e23a6ab6590a","Type":"ContainerStarted","Data":"a9796911ccd9bb83300c87308df98592b5b8559d450ac2543ff79f805648e290"} Nov 03 21:10:33.396533 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:33.396486 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-1e5494-5599ddf75f-b2h97" Nov 03 21:10:33.397942 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:33.397918 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-1e5494-7cd9cc6ffd-jqr7r" event={"ID":"dc0be43c-23aa-4e0c-b69a-34162e3fa0db","Type":"ContainerStarted","Data":"ef7dd8c890e58de3f9bfe1cbe49ebdde5544ee9562c388bb363c952d9587d353"} Nov 03 21:10:33.398084 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:33.397955 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-1e5494-7cd9cc6ffd-jqr7r" event={"ID":"dc0be43c-23aa-4e0c-b69a-34162e3fa0db","Type":"ContainerStarted","Data":"43efc4e022c0589640d0f2363135584a2cc01592c044ca0e2af8fc75c494acdd"} Nov 03 21:10:33.398084 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:33.398005 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-1e5494-7cd9cc6ffd-jqr7r" Nov 03 21:10:33.514598 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:33.514529 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-1e5494-5599ddf75f-b2h97" podStartSLOduration=2.5145122840000003 podStartE2EDuration="2.514512284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:10:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:10:33.511239374 +0000 UTC m=+1868.736559165" watchObservedRunningTime="2026-11-03 21:10:33.514512284 +0000 UTC m=+1868.739832076" Nov 03 21:10:33.631365 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:33.631307 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-1e5494-7cd9cc6ffd-jqr7r" podStartSLOduration=2.631287956 podStartE2EDuration="2.631287956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:10:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:10:33.626539613 +0000 UTC m=+1868.851859432" watchObservedRunningTime="2026-11-03 21:10:33.631287956 +0000 UTC m=+1868.856607746" Nov 03 21:10:37.347269 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:37.347239 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-32c41e-57d86546c-26fn2" Nov 03 21:10:38.358309 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:38.358271 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-32c41e-794b58b44-25fvr" Nov 03 21:10:42.451743 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:42.451709 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-206508-9hwdl"] Nov 03 21:10:42.456202 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:42.456174 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-206508-9hwdl" Nov 03 21:10:42.514004 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:42.513967 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-206508-9hwdl"] Nov 03 21:10:42.557183 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:42.557146 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:10:42.557183 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:42.557157 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:10:42.557510 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:42.557238 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:10:42.557510 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:42.557383 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77vtd\"" Nov 03 21:10:42.607769 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:42.607735 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/964b98c5-2a4e-41e8-b707-4f2915e61933-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-206508-9hwdl\" (UID: \"964b98c5-2a4e-41e8-b707-4f2915e61933\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-206508-9hwdl" Nov 03 21:10:42.607955 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:42.607894 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h7gj\" (UniqueName: \"kubernetes.io/projected/964b98c5-2a4e-41e8-b707-4f2915e61933-kube-api-access-2h7gj\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-206508-9hwdl\" (UID: \"964b98c5-2a4e-41e8-b707-4f2915e61933\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-206508-9hwdl" Nov 03 21:10:42.608037 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:42.608022 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/964b98c5-2a4e-41e8-b707-4f2915e61933-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-206508-9hwdl\" (UID: \"964b98c5-2a4e-41e8-b707-4f2915e61933\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-206508-9hwdl" Nov 03 21:10:42.709251 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:42.709154 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/964b98c5-2a4e-41e8-b707-4f2915e61933-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-206508-9hwdl\" (UID: \"964b98c5-2a4e-41e8-b707-4f2915e61933\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-206508-9hwdl" Nov 03 21:10:42.709251 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:42.709244 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/964b98c5-2a4e-41e8-b707-4f2915e61933-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-206508-9hwdl\" (UID: \"964b98c5-2a4e-41e8-b707-4f2915e61933\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-206508-9hwdl" Nov 03 21:10:42.709562 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:42.709319 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h7gj\" (UniqueName: \"kubernetes.io/projected/964b98c5-2a4e-41e8-b707-4f2915e61933-kube-api-access-2h7gj\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-206508-9hwdl\" (UID: \"964b98c5-2a4e-41e8-b707-4f2915e61933\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-206508-9hwdl" Nov 03 21:10:42.710065 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:42.710039 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/964b98c5-2a4e-41e8-b707-4f2915e61933-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-206508-9hwdl\" (UID: \"964b98c5-2a4e-41e8-b707-4f2915e61933\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-206508-9hwdl" Nov 03 21:10:42.712032 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:42.712007 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/964b98c5-2a4e-41e8-b707-4f2915e61933-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-206508-9hwdl\" (UID: \"964b98c5-2a4e-41e8-b707-4f2915e61933\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-206508-9hwdl" Nov 03 21:10:42.772988 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:42.772957 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h7gj\" (UniqueName: \"kubernetes.io/projected/964b98c5-2a4e-41e8-b707-4f2915e61933-kube-api-access-2h7gj\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-206508-9hwdl\" (UID: \"964b98c5-2a4e-41e8-b707-4f2915e61933\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-206508-9hwdl" Nov 03 21:10:43.049540 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:43.049415 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8f1dfd9-wpnkr"] Nov 03 21:10:43.053124 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:43.053100 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8f1dfd9-wpnkr"] Nov 03 21:10:43.072363 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:43.072333 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-206508-9hwdl" Nov 03 21:10:43.454929 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:43.454895 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-206508-9hwdl"] Nov 03 21:10:43.456190 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:10:43.456159 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod964b98c5_2a4e_41e8_b707_4f2915e61933.slice/crio-fb6949410146d9fa839eb09e8365510b592da5c48d2a0f7f97d8ceb7aea64f07 WatchSource:0}: Error finding container fb6949410146d9fa839eb09e8365510b592da5c48d2a0f7f97d8ceb7aea64f07: Status 404 returned error can't find the container with id fb6949410146d9fa839eb09e8365510b592da5c48d2a0f7f97d8ceb7aea64f07 Nov 03 21:10:43.474759 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:43.474726 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-206508-9hwdl" event={"ID":"964b98c5-2a4e-41e8-b707-4f2915e61933","Type":"ContainerStarted","Data":"fb6949410146d9fa839eb09e8365510b592da5c48d2a0f7f97d8ceb7aea64f07"} Nov 03 21:10:44.408645 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:44.408615 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-1e5494-7cd9cc6ffd-jqr7r" Nov 03 21:10:44.408855 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:44.408837 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-1e5494-5599ddf75f-b2h97" Nov 03 21:10:44.484567 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:44.484534 4305 generic.go:356] "Generic (PLEG): container finished" podID="964b98c5-2a4e-41e8-b707-4f2915e61933" containerID="f7634e00dc13bd882cc5d5fe561e6dedff6f8ff3cfa6ad82d25a69c144ab1efb" exitCode=0 Nov 03 21:10:44.484929 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:44.484601 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-206508-9hwdl" event={"ID":"964b98c5-2a4e-41e8-b707-4f2915e61933","Type":"ContainerDied","Data":"f7634e00dc13bd882cc5d5fe561e6dedff6f8ff3cfa6ad82d25a69c144ab1efb"} Nov 03 21:10:44.896238 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:44.896190 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="485058fd-0a87-4d4c-aaa0-bfef9ceb39ef" path="/var/lib/kubelet/pods/485058fd-0a87-4d4c-aaa0-bfef9ceb39ef/volumes" Nov 03 21:10:45.659937 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:45.659904 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-206508-9hwdl" Nov 03 21:10:45.751707 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:45.751616 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/964b98c5-2a4e-41e8-b707-4f2915e61933-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/964b98c5-2a4e-41e8-b707-4f2915e61933-maas-api-openshift-ca\") pod \"964b98c5-2a4e-41e8-b707-4f2915e61933\" (UID: \"964b98c5-2a4e-41e8-b707-4f2915e61933\") " Nov 03 21:10:45.751903 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:45.751825 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/964b98c5-2a4e-41e8-b707-4f2915e61933-kube-api-access-2h7gj\" (UniqueName: \"kubernetes.io/projected/964b98c5-2a4e-41e8-b707-4f2915e61933-kube-api-access-2h7gj\") pod \"964b98c5-2a4e-41e8-b707-4f2915e61933\" (UID: \"964b98c5-2a4e-41e8-b707-4f2915e61933\") " Nov 03 21:10:45.751976 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:45.751945 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/964b98c5-2a4e-41e8-b707-4f2915e61933-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/964b98c5-2a4e-41e8-b707-4f2915e61933-maas-api-xks-ca\") pod \"964b98c5-2a4e-41e8-b707-4f2915e61933\" (UID: \"964b98c5-2a4e-41e8-b707-4f2915e61933\") " Nov 03 21:10:45.752073 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:45.752042 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/964b98c5-2a4e-41e8-b707-4f2915e61933-maas-api-openshift-ca" pod "964b98c5-2a4e-41e8-b707-4f2915e61933" (UID: "964b98c5-2a4e-41e8-b707-4f2915e61933"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:10:45.752776 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:45.752748 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/964b98c5-2a4e-41e8-b707-4f2915e61933-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:10:45.754213 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:45.754184 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/964b98c5-2a4e-41e8-b707-4f2915e61933-maas-api-xks-ca" pod "964b98c5-2a4e-41e8-b707-4f2915e61933" (UID: "964b98c5-2a4e-41e8-b707-4f2915e61933"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:10:45.754341 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:45.754295 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/964b98c5-2a4e-41e8-b707-4f2915e61933-kube-api-access-2h7gj" pod "964b98c5-2a4e-41e8-b707-4f2915e61933" (UID: "964b98c5-2a4e-41e8-b707-4f2915e61933"). InnerVolumeSpecName "kube-api-access-2h7gj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:10:45.854466 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:45.854396 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/964b98c5-2a4e-41e8-b707-4f2915e61933-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:10:45.854466 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:45.854428 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h7gj\" (UniqueName: \"kubernetes.io/projected/964b98c5-2a4e-41e8-b707-4f2915e61933-kube-api-access-2h7gj\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:10:46.324218 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:10:46.324178 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:10:46.499465 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:46.499417 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-206508-9hwdl" Nov 03 21:10:46.499653 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:46.499472 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-206508-9hwdl" event={"ID":"964b98c5-2a4e-41e8-b707-4f2915e61933","Type":"ContainerDied","Data":"fb6949410146d9fa839eb09e8365510b592da5c48d2a0f7f97d8ceb7aea64f07"} Nov 03 21:10:46.499653 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:46.499521 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb6949410146d9fa839eb09e8365510b592da5c48d2a0f7f97d8ceb7aea64f07" Nov 03 21:10:47.781899 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:47.781865 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v"] Nov 03 21:10:47.782640 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:47.782623 4305 cpu_manager.go:414] "containerMap: removing container" podUID="964b98c5-2a4e-41e8-b707-4f2915e61933" containerName="revoke-keys" Nov 03 21:10:47.782701 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:47.782641 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="964b98c5-2a4e-41e8-b707-4f2915e61933" containerName="revoke-keys" Nov 03 21:10:47.782770 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:47.782761 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="964b98c5-2a4e-41e8-b707-4f2915e61933" containerName="revoke-keys" Nov 03 21:10:47.786273 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:47.786253 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v" Nov 03 21:10:47.850231 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:47.850197 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v"] Nov 03 21:10:47.885392 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:47.885358 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/239c6c9d-3ac5-49b5-9093-e686f002ce2a-workload-socket\") pod \"e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v\" (UID: \"239c6c9d-3ac5-49b5-9093-e686f002ce2a\") " pod="openshift-ingress/e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v" Nov 03 21:10:47.885590 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:47.885468 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/239c6c9d-3ac5-49b5-9093-e686f002ce2a-credential-socket\") pod \"e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v\" (UID: \"239c6c9d-3ac5-49b5-9093-e686f002ce2a\") " pod="openshift-ingress/e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v" Nov 03 21:10:47.885637 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:47.885603 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5px9d\" (UniqueName: \"kubernetes.io/projected/239c6c9d-3ac5-49b5-9093-e686f002ce2a-kube-api-access-5px9d\") pod \"e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v\" (UID: \"239c6c9d-3ac5-49b5-9093-e686f002ce2a\") " pod="openshift-ingress/e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v" Nov 03 21:10:47.885697 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:47.885654 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/239c6c9d-3ac5-49b5-9093-e686f002ce2a-istio-data\") pod \"e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v\" (UID: \"239c6c9d-3ac5-49b5-9093-e686f002ce2a\") " pod="openshift-ingress/e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v" Nov 03 21:10:47.885768 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:47.885754 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/239c6c9d-3ac5-49b5-9093-e686f002ce2a-istio-envoy\") pod \"e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v\" (UID: \"239c6c9d-3ac5-49b5-9093-e686f002ce2a\") " pod="openshift-ingress/e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v" Nov 03 21:10:47.885825 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:47.885778 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/239c6c9d-3ac5-49b5-9093-e686f002ce2a-istiod-ca-cert\") pod \"e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v\" (UID: \"239c6c9d-3ac5-49b5-9093-e686f002ce2a\") " pod="openshift-ingress/e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v" Nov 03 21:10:47.885873 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:47.885844 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/239c6c9d-3ac5-49b5-9093-e686f002ce2a-istio-podinfo\") pod \"e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v\" (UID: \"239c6c9d-3ac5-49b5-9093-e686f002ce2a\") " pod="openshift-ingress/e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v" Nov 03 21:10:47.885919 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:47.885874 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/239c6c9d-3ac5-49b5-9093-e686f002ce2a-istio-token\") pod \"e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v\" (UID: \"239c6c9d-3ac5-49b5-9093-e686f002ce2a\") " pod="openshift-ingress/e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v" Nov 03 21:10:47.885952 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:47.885916 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/239c6c9d-3ac5-49b5-9093-e686f002ce2a-workload-certs\") pod \"e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v\" (UID: \"239c6c9d-3ac5-49b5-9093-e686f002ce2a\") " pod="openshift-ingress/e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v" Nov 03 21:10:47.899517 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:47.899491 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-01c7a3-openshift-default-dockercfg-w68bb\"" Nov 03 21:10:47.987668 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:47.987626 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/239c6c9d-3ac5-49b5-9093-e686f002ce2a-istio-podinfo\") pod \"e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v\" (UID: \"239c6c9d-3ac5-49b5-9093-e686f002ce2a\") " pod="openshift-ingress/e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v" Nov 03 21:10:47.988413 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:47.988387 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/239c6c9d-3ac5-49b5-9093-e686f002ce2a-istio-token\") pod \"e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v\" (UID: \"239c6c9d-3ac5-49b5-9093-e686f002ce2a\") " pod="openshift-ingress/e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v" Nov 03 21:10:47.988972 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:47.988952 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/239c6c9d-3ac5-49b5-9093-e686f002ce2a-workload-certs\") pod \"e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v\" (UID: \"239c6c9d-3ac5-49b5-9093-e686f002ce2a\") " pod="openshift-ingress/e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v" Nov 03 21:10:47.989551 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:47.989410 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/239c6c9d-3ac5-49b5-9093-e686f002ce2a-workload-certs\") pod \"e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v\" (UID: \"239c6c9d-3ac5-49b5-9093-e686f002ce2a\") " pod="openshift-ingress/e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v" Nov 03 21:10:47.989803 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:47.989781 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/239c6c9d-3ac5-49b5-9093-e686f002ce2a-workload-socket\") pod \"e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v\" (UID: \"239c6c9d-3ac5-49b5-9093-e686f002ce2a\") " pod="openshift-ingress/e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v" Nov 03 21:10:47.998416 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:47.998388 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/239c6c9d-3ac5-49b5-9093-e686f002ce2a-credential-socket\") pod \"e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v\" (UID: \"239c6c9d-3ac5-49b5-9093-e686f002ce2a\") " pod="openshift-ingress/e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v" Nov 03 21:10:47.999511 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:47.999486 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5px9d\" (UniqueName: \"kubernetes.io/projected/239c6c9d-3ac5-49b5-9093-e686f002ce2a-kube-api-access-5px9d\") pod \"e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v\" (UID: \"239c6c9d-3ac5-49b5-9093-e686f002ce2a\") " pod="openshift-ingress/e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v" Nov 03 21:10:48.000054 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:48.000034 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/239c6c9d-3ac5-49b5-9093-e686f002ce2a-istio-data\") pod \"e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v\" (UID: \"239c6c9d-3ac5-49b5-9093-e686f002ce2a\") " pod="openshift-ingress/e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v" Nov 03 21:10:48.000745 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:48.000726 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/239c6c9d-3ac5-49b5-9093-e686f002ce2a-istio-envoy\") pod \"e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v\" (UID: \"239c6c9d-3ac5-49b5-9093-e686f002ce2a\") " pod="openshift-ingress/e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v" Nov 03 21:10:48.001594 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:48.001576 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/239c6c9d-3ac5-49b5-9093-e686f002ce2a-istiod-ca-cert\") pod \"e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v\" (UID: \"239c6c9d-3ac5-49b5-9093-e686f002ce2a\") " pod="openshift-ingress/e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v" Nov 03 21:10:48.002467 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:47.997198 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/239c6c9d-3ac5-49b5-9093-e686f002ce2a-workload-socket\") pod \"e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v\" (UID: \"239c6c9d-3ac5-49b5-9093-e686f002ce2a\") " pod="openshift-ingress/e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v" Nov 03 21:10:48.002615 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:48.001517 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/239c6c9d-3ac5-49b5-9093-e686f002ce2a-istio-podinfo\") pod \"e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v\" (UID: \"239c6c9d-3ac5-49b5-9093-e686f002ce2a\") " pod="openshift-ingress/e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v" Nov 03 21:10:48.002719 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:47.999329 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/239c6c9d-3ac5-49b5-9093-e686f002ce2a-credential-socket\") pod \"e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v\" (UID: \"239c6c9d-3ac5-49b5-9093-e686f002ce2a\") " pod="openshift-ingress/e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v" Nov 03 21:10:48.002811 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:48.000621 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/239c6c9d-3ac5-49b5-9093-e686f002ce2a-istio-data\") pod \"e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v\" (UID: \"239c6c9d-3ac5-49b5-9093-e686f002ce2a\") " pod="openshift-ingress/e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v" Nov 03 21:10:48.002909 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:48.002305 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/239c6c9d-3ac5-49b5-9093-e686f002ce2a-istiod-ca-cert\") pod \"e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v\" (UID: \"239c6c9d-3ac5-49b5-9093-e686f002ce2a\") " pod="openshift-ingress/e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v" Nov 03 21:10:48.015465 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:48.015413 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ca4642-6dsn2"] Nov 03 21:10:48.026551 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:48.026491 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-ca4642-6dsn2"] Nov 03 21:10:48.027654 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:48.027626 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/239c6c9d-3ac5-49b5-9093-e686f002ce2a-istio-envoy\") pod \"e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v\" (UID: \"239c6c9d-3ac5-49b5-9093-e686f002ce2a\") " pod="openshift-ingress/e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v" Nov 03 21:10:48.063623 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:48.063006 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/239c6c9d-3ac5-49b5-9093-e686f002ce2a-istio-token\") pod \"e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v\" (UID: \"239c6c9d-3ac5-49b5-9093-e686f002ce2a\") " pod="openshift-ingress/e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v" Nov 03 21:10:48.074485 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:48.073462 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5px9d\" (UniqueName: \"kubernetes.io/projected/239c6c9d-3ac5-49b5-9093-e686f002ce2a-kube-api-access-5px9d\") pod \"e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v\" (UID: \"239c6c9d-3ac5-49b5-9093-e686f002ce2a\") " pod="openshift-ingress/e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v" Nov 03 21:10:48.102156 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:48.102118 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v" Nov 03 21:10:48.551613 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:48.551582 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v"] Nov 03 21:10:48.552266 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:10:48.552241 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod239c6c9d_3ac5_49b5_9093_e686f002ce2a.slice/crio-9d91e4c52012709bd105882b8a2cff3b0a018496cfac47e1f4d50298ed413ba2 WatchSource:0}: Error finding container 9d91e4c52012709bd105882b8a2cff3b0a018496cfac47e1f4d50298ed413ba2: Status 404 returned error can't find the container with id 9d91e4c52012709bd105882b8a2cff3b0a018496cfac47e1f4d50298ed413ba2 Nov 03 21:10:48.555168 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:48.555128 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:10:48.555233 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:48.555220 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:10:48.555278 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:48.555268 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:10:48.896428 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:48.896398 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="559f3f98-6c70-42b2-aa59-22cb1902e41d" path="/var/lib/kubelet/pods/559f3f98-6c70-42b2-aa59-22cb1902e41d/volumes" Nov 03 21:10:49.523088 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:49.523045 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v" event={"ID":"239c6c9d-3ac5-49b5-9093-e686f002ce2a","Type":"ContainerStarted","Data":"0eeba93b3f6bc986d8eda7e3d9427349d5e1c7704233e90443cdd1a54dd67d1d"} Nov 03 21:10:49.523288 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:49.523113 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v" event={"ID":"239c6c9d-3ac5-49b5-9093-e686f002ce2a","Type":"ContainerStarted","Data":"9d91e4c52012709bd105882b8a2cff3b0a018496cfac47e1f4d50298ed413ba2"} Nov 03 21:10:49.746735 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:49.746687 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v" podStartSLOduration=2.7466717000000003 podStartE2EDuration="2.7466717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:10:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:10:49.692871779 +0000 UTC m=+1884.918191591" watchObservedRunningTime="2026-11-03 21:10:49.7466717 +0000 UTC m=+1884.971991491" Nov 03 21:10:50.102949 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:50.102911 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v" Nov 03 21:10:50.109183 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:50.109156 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v" Nov 03 21:10:50.530490 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:50.530460 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v" Nov 03 21:10:50.531639 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:50.531617 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v" Nov 03 21:10:55.696901 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:55.696857 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9"] Nov 03 21:10:55.697579 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:55.697174 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9" podUID="9f8328f2-ffba-400c-b4e3-c6c62092fefd" containerName="istio-proxy" containerID="cri-o://bf55d75d372a4c7e620a0ecd4e1abb1258da7cadad6085e78f6fea556ece66bb" gracePeriod=30 Nov 03 21:10:56.596564 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:56.593410 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft"] Nov 03 21:10:56.596564 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:56.593778 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft" podUID="365e3804-37c5-4398-95c0-a1602237453e" containerName="istio-proxy" containerID="cri-o://097e2b681c7777e9b661d86817933bb1d1b6cbd767c560ff92ea59a4807a981a" gracePeriod=30 Nov 03 21:10:58.415422 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:58.415376 4305 patch_prober.go:29] interesting pod/e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:10:58.415853 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:58.415472 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9" podUID="9f8328f2-ffba-400c-b4e3-c6c62092fefd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:10:59.431088 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:59.431054 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01c7a3-6479dcc9-xzqvp"] Nov 03 21:10:59.436896 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:59.436877 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01c7a3-6479dcc9-xzqvp" Nov 03 21:10:59.489225 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:59.489194 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-01c7a3\"" Nov 03 21:10:59.492929 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:59.492899 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01c7a3-6479dcc9-xzqvp"] Nov 03 21:10:59.497329 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:59.497300 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eff346af-88ef-494d-8505-c8546c5c6f4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-01c7a3-6479dcc9-xzqvp\" (UID: \"eff346af-88ef-494d-8505-c8546c5c6f4e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01c7a3-6479dcc9-xzqvp" Nov 03 21:10:59.497492 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:59.497343 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t64zd\" (UniqueName: \"kubernetes.io/projected/eff346af-88ef-494d-8505-c8546c5c6f4e-kube-api-access-t64zd\") pod \"payload-pre-processing-e2e-shared-b-w2-01c7a3-6479dcc9-xzqvp\" (UID: \"eff346af-88ef-494d-8505-c8546c5c6f4e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01c7a3-6479dcc9-xzqvp" Nov 03 21:10:59.594461 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:59.589177 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-01c7a3-dockercfg-jwk9v\"" Nov 03 21:10:59.631459 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:59.611783 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eff346af-88ef-494d-8505-c8546c5c6f4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-01c7a3-6479dcc9-xzqvp\" (UID: \"eff346af-88ef-494d-8505-c8546c5c6f4e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01c7a3-6479dcc9-xzqvp" Nov 03 21:10:59.648454 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:59.648387 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t64zd\" (UniqueName: \"kubernetes.io/projected/eff346af-88ef-494d-8505-c8546c5c6f4e-kube-api-access-t64zd\") pod \"payload-pre-processing-e2e-shared-b-w2-01c7a3-6479dcc9-xzqvp\" (UID: \"eff346af-88ef-494d-8505-c8546c5c6f4e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01c7a3-6479dcc9-xzqvp" Nov 03 21:10:59.656557 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:59.656520 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eff346af-88ef-494d-8505-c8546c5c6f4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-01c7a3-6479dcc9-xzqvp\" (UID: \"eff346af-88ef-494d-8505-c8546c5c6f4e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01c7a3-6479dcc9-xzqvp" Nov 03 21:10:59.664543 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:59.662297 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-01c7a3-55bcd6997b-k8qhc"] Nov 03 21:10:59.682615 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:59.682539 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-01c7a3-55bcd6997b-k8qhc"] Nov 03 21:10:59.682916 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:59.682901 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-01c7a3-55bcd6997b-k8qhc" Nov 03 21:10:59.718130 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:59.718095 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t64zd\" (UniqueName: \"kubernetes.io/projected/eff346af-88ef-494d-8505-c8546c5c6f4e-kube-api-access-t64zd\") pod \"payload-pre-processing-e2e-shared-b-w2-01c7a3-6479dcc9-xzqvp\" (UID: \"eff346af-88ef-494d-8505-c8546c5c6f4e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01c7a3-6479dcc9-xzqvp" Nov 03 21:10:59.751872 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:59.751841 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01c7a3-6479dcc9-xzqvp" Nov 03 21:10:59.756252 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:59.756227 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4t6b\" (UniqueName: \"kubernetes.io/projected/7fb81be4-4690-45e1-947f-2de9f107d4d7-kube-api-access-n4t6b\") pod \"payload-processing-e2e-shared-b-w2-01c7a3-55bcd6997b-k8qhc\" (UID: \"7fb81be4-4690-45e1-947f-2de9f107d4d7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-01c7a3-55bcd6997b-k8qhc" Nov 03 21:10:59.756538 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:59.756422 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fb81be4-4690-45e1-947f-2de9f107d4d7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-01c7a3-55bcd6997b-k8qhc\" (UID: \"7fb81be4-4690-45e1-947f-2de9f107d4d7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-01c7a3-55bcd6997b-k8qhc" Nov 03 21:10:59.857840 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:59.857808 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fb81be4-4690-45e1-947f-2de9f107d4d7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-01c7a3-55bcd6997b-k8qhc\" (UID: \"7fb81be4-4690-45e1-947f-2de9f107d4d7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-01c7a3-55bcd6997b-k8qhc" Nov 03 21:10:59.858022 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:59.857936 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4t6b\" (UniqueName: \"kubernetes.io/projected/7fb81be4-4690-45e1-947f-2de9f107d4d7-kube-api-access-n4t6b\") pod \"payload-processing-e2e-shared-b-w2-01c7a3-55bcd6997b-k8qhc\" (UID: \"7fb81be4-4690-45e1-947f-2de9f107d4d7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-01c7a3-55bcd6997b-k8qhc" Nov 03 21:10:59.859205 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:59.859137 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fb81be4-4690-45e1-947f-2de9f107d4d7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-01c7a3-55bcd6997b-k8qhc\" (UID: \"7fb81be4-4690-45e1-947f-2de9f107d4d7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-01c7a3-55bcd6997b-k8qhc" Nov 03 21:10:59.919028 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:10:59.918993 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4t6b\" (UniqueName: \"kubernetes.io/projected/7fb81be4-4690-45e1-947f-2de9f107d4d7-kube-api-access-n4t6b\") pod \"payload-processing-e2e-shared-b-w2-01c7a3-55bcd6997b-k8qhc\" (UID: \"7fb81be4-4690-45e1-947f-2de9f107d4d7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-01c7a3-55bcd6997b-k8qhc" Nov 03 21:11:00.016592 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:00.016516 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-01c7a3-55bcd6997b-k8qhc" Nov 03 21:11:00.139701 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:00.139668 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01c7a3-6479dcc9-xzqvp"] Nov 03 21:11:00.397562 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:00.397526 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-01c7a3-55bcd6997b-k8qhc"] Nov 03 21:11:00.696799 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:00.696708 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01c7a3-6479dcc9-xzqvp" event={"ID":"eff346af-88ef-494d-8505-c8546c5c6f4e","Type":"ContainerStarted","Data":"2f6806542fb8a8a58fe50629a9b9615be05bc76c89cd6ce0da633a5ff581aed9"} Nov 03 21:11:00.697285 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:00.696811 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01c7a3-6479dcc9-xzqvp" event={"ID":"eff346af-88ef-494d-8505-c8546c5c6f4e","Type":"ContainerStarted","Data":"042dc699f618e329bd7c89aef947bcc7173a1880399ec5c96c1a79b6c82da3fb"} Nov 03 21:11:00.697285 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:00.696950 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01c7a3-6479dcc9-xzqvp" Nov 03 21:11:00.698536 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:00.698513 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-01c7a3-55bcd6997b-k8qhc" event={"ID":"7fb81be4-4690-45e1-947f-2de9f107d4d7","Type":"ContainerStarted","Data":"39f3dd56347f4b50bdc9ddf83440d99d31ff1da3aab741b59af6b8fb62e7cb87"} Nov 03 21:11:00.698640 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:00.698550 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-01c7a3-55bcd6997b-k8qhc" event={"ID":"7fb81be4-4690-45e1-947f-2de9f107d4d7","Type":"ContainerStarted","Data":"91e7a462b8c819964c9739c2128f269f8c3f78efba174227ed73e47ef5cb1ef7"} Nov 03 21:11:00.698683 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:00.698656 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-01c7a3-55bcd6997b-k8qhc" Nov 03 21:11:00.816522 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:00.816470 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01c7a3-6479dcc9-xzqvp" podStartSLOduration=1.816454459 podStartE2EDuration="1.816454459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:10:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:11:00.811881491 +0000 UTC m=+1896.037201284" watchObservedRunningTime="2026-11-03 21:11:00.816454459 +0000 UTC m=+1896.041774244" Nov 03 21:11:00.946899 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:00.946785 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-01c7a3-55bcd6997b-k8qhc" podStartSLOduration=1.9467633100000001 podStartE2EDuration="1.94676331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:10:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:11:00.933496129 +0000 UTC m=+1896.158815921" watchObservedRunningTime="2026-11-03 21:11:00.94676331 +0000 UTC m=+1896.172083107" Nov 03 21:11:01.011958 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:01.011930 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9" Nov 03 21:11:01.072362 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:01.072332 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f8328f2-ffba-400c-b4e3-c6c62092fefd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f8328f2-ffba-400c-b4e3-c6c62092fefd-istiod-ca-cert\") pod \"9f8328f2-ffba-400c-b4e3-c6c62092fefd\" (UID: \"9f8328f2-ffba-400c-b4e3-c6c62092fefd\") " Nov 03 21:11:01.072596 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:01.072372 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f8328f2-ffba-400c-b4e3-c6c62092fefd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f8328f2-ffba-400c-b4e3-c6c62092fefd-workload-certs\") pod \"9f8328f2-ffba-400c-b4e3-c6c62092fefd\" (UID: \"9f8328f2-ffba-400c-b4e3-c6c62092fefd\") " Nov 03 21:11:01.072596 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:01.072572 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f8328f2-ffba-400c-b4e3-c6c62092fefd-istio-token\" (UniqueName: \"kubernetes.io/projected/9f8328f2-ffba-400c-b4e3-c6c62092fefd-istio-token\") pod \"9f8328f2-ffba-400c-b4e3-c6c62092fefd\" (UID: \"9f8328f2-ffba-400c-b4e3-c6c62092fefd\") " Nov 03 21:11:01.072743 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:01.072600 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f8328f2-ffba-400c-b4e3-c6c62092fefd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f8328f2-ffba-400c-b4e3-c6c62092fefd-istio-podinfo\") pod \"9f8328f2-ffba-400c-b4e3-c6c62092fefd\" (UID: \"9f8328f2-ffba-400c-b4e3-c6c62092fefd\") " Nov 03 21:11:01.072743 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:01.072647 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f8328f2-ffba-400c-b4e3-c6c62092fefd-kube-api-access-nl425\" (UniqueName: \"kubernetes.io/projected/9f8328f2-ffba-400c-b4e3-c6c62092fefd-kube-api-access-nl425\") pod \"9f8328f2-ffba-400c-b4e3-c6c62092fefd\" (UID: \"9f8328f2-ffba-400c-b4e3-c6c62092fefd\") " Nov 03 21:11:01.072869 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:01.072750 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f8328f2-ffba-400c-b4e3-c6c62092fefd-workload-certs" pod "9f8328f2-ffba-400c-b4e3-c6c62092fefd" (UID: "9f8328f2-ffba-400c-b4e3-c6c62092fefd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:11:01.072869 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:01.072774 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f8328f2-ffba-400c-b4e3-c6c62092fefd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f8328f2-ffba-400c-b4e3-c6c62092fefd-istio-envoy\") pod \"9f8328f2-ffba-400c-b4e3-c6c62092fefd\" (UID: \"9f8328f2-ffba-400c-b4e3-c6c62092fefd\") " Nov 03 21:11:01.072869 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:01.072788 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f8328f2-ffba-400c-b4e3-c6c62092fefd-istiod-ca-cert" pod "9f8328f2-ffba-400c-b4e3-c6c62092fefd" (UID: "9f8328f2-ffba-400c-b4e3-c6c62092fefd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:11:01.073043 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:01.072882 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f8328f2-ffba-400c-b4e3-c6c62092fefd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f8328f2-ffba-400c-b4e3-c6c62092fefd-workload-socket\") pod \"9f8328f2-ffba-400c-b4e3-c6c62092fefd\" (UID: \"9f8328f2-ffba-400c-b4e3-c6c62092fefd\") " Nov 03 21:11:01.073043 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:01.072929 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f8328f2-ffba-400c-b4e3-c6c62092fefd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f8328f2-ffba-400c-b4e3-c6c62092fefd-credential-socket\") pod \"9f8328f2-ffba-400c-b4e3-c6c62092fefd\" (UID: \"9f8328f2-ffba-400c-b4e3-c6c62092fefd\") " Nov 03 21:11:01.073043 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:01.072980 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f8328f2-ffba-400c-b4e3-c6c62092fefd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f8328f2-ffba-400c-b4e3-c6c62092fefd-istio-data\") pod \"9f8328f2-ffba-400c-b4e3-c6c62092fefd\" (UID: \"9f8328f2-ffba-400c-b4e3-c6c62092fefd\") " Nov 03 21:11:01.073251 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:01.073230 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f8328f2-ffba-400c-b4e3-c6c62092fefd-workload-socket" pod "9f8328f2-ffba-400c-b4e3-c6c62092fefd" (UID: "9f8328f2-ffba-400c-b4e3-c6c62092fefd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:11:01.073611 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:01.073409 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f8328f2-ffba-400c-b4e3-c6c62092fefd-istio-data" pod "9f8328f2-ffba-400c-b4e3-c6c62092fefd" (UID: "9f8328f2-ffba-400c-b4e3-c6c62092fefd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:11:01.073611 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:01.073589 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f8328f2-ffba-400c-b4e3-c6c62092fefd-credential-socket" pod "9f8328f2-ffba-400c-b4e3-c6c62092fefd" (UID: "9f8328f2-ffba-400c-b4e3-c6c62092fefd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:11:01.073782 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:01.073720 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f8328f2-ffba-400c-b4e3-c6c62092fefd-workload-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:11:01.073782 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:01.073734 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f8328f2-ffba-400c-b4e3-c6c62092fefd-credential-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:11:01.073782 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:01.073744 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f8328f2-ffba-400c-b4e3-c6c62092fefd-istio-data\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:11:01.073782 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:01.073753 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f8328f2-ffba-400c-b4e3-c6c62092fefd-istiod-ca-cert\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:11:01.073782 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:01.073760 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f8328f2-ffba-400c-b4e3-c6c62092fefd-workload-certs\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:11:01.075271 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:01.075246 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f8328f2-ffba-400c-b4e3-c6c62092fefd-istio-token" pod "9f8328f2-ffba-400c-b4e3-c6c62092fefd" (UID: "9f8328f2-ffba-400c-b4e3-c6c62092fefd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:11:01.075378 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:01.075291 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f8328f2-ffba-400c-b4e3-c6c62092fefd-kube-api-access-nl425" pod "9f8328f2-ffba-400c-b4e3-c6c62092fefd" (UID: "9f8328f2-ffba-400c-b4e3-c6c62092fefd"). InnerVolumeSpecName "kube-api-access-nl425". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:11:01.075378 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:01.075347 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f8328f2-ffba-400c-b4e3-c6c62092fefd-istio-podinfo" pod "9f8328f2-ffba-400c-b4e3-c6c62092fefd" (UID: "9f8328f2-ffba-400c-b4e3-c6c62092fefd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:11:01.075505 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:01.075419 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f8328f2-ffba-400c-b4e3-c6c62092fefd-istio-envoy" pod "9f8328f2-ffba-400c-b4e3-c6c62092fefd" (UID: "9f8328f2-ffba-400c-b4e3-c6c62092fefd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:11:01.175749 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:01.175710 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f8328f2-ffba-400c-b4e3-c6c62092fefd-istio-token\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:11:01.175749 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:01.175750 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f8328f2-ffba-400c-b4e3-c6c62092fefd-istio-podinfo\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:11:01.175981 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:01.175767 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nl425\" (UniqueName: \"kubernetes.io/projected/9f8328f2-ffba-400c-b4e3-c6c62092fefd-kube-api-access-nl425\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:11:01.175981 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:01.175784 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f8328f2-ffba-400c-b4e3-c6c62092fefd-istio-envoy\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:11:01.707025 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:01.706990 4305 generic.go:356] "Generic (PLEG): container finished" podID="9f8328f2-ffba-400c-b4e3-c6c62092fefd" containerID="bf55d75d372a4c7e620a0ecd4e1abb1258da7cadad6085e78f6fea556ece66bb" exitCode=0 Nov 03 21:11:01.707562 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:01.707054 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9" Nov 03 21:11:01.707562 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:01.707101 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9" event={"ID":"9f8328f2-ffba-400c-b4e3-c6c62092fefd","Type":"ContainerDied","Data":"bf55d75d372a4c7e620a0ecd4e1abb1258da7cadad6085e78f6fea556ece66bb"} Nov 03 21:11:01.707562 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:01.707191 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9" event={"ID":"9f8328f2-ffba-400c-b4e3-c6c62092fefd","Type":"ContainerDied","Data":"71e87e5e3771fc21d13a28ce880d4240e6df532d606eccf084ea75040cd67fb9"} Nov 03 21:11:01.707562 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:01.707214 4305 scope.go:122] "RemoveContainer" containerID="bf55d75d372a4c7e620a0ecd4e1abb1258da7cadad6085e78f6fea556ece66bb" Nov 03 21:11:01.718418 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:01.718391 4305 scope.go:122] "RemoveContainer" containerID="bf55d75d372a4c7e620a0ecd4e1abb1258da7cadad6085e78f6fea556ece66bb" Nov 03 21:11:01.718902 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:11:01.718884 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf55d75d372a4c7e620a0ecd4e1abb1258da7cadad6085e78f6fea556ece66bb\": container with ID starting with bf55d75d372a4c7e620a0ecd4e1abb1258da7cadad6085e78f6fea556ece66bb not found: ID does not exist" containerID="bf55d75d372a4c7e620a0ecd4e1abb1258da7cadad6085e78f6fea556ece66bb" Nov 03 21:11:01.718976 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:01.718913 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf55d75d372a4c7e620a0ecd4e1abb1258da7cadad6085e78f6fea556ece66bb"} err="failed to get container status \"bf55d75d372a4c7e620a0ecd4e1abb1258da7cadad6085e78f6fea556ece66bb\": rpc error: code = NotFound desc = could not find container \"bf55d75d372a4c7e620a0ecd4e1abb1258da7cadad6085e78f6fea556ece66bb\": container with ID starting with bf55d75d372a4c7e620a0ecd4e1abb1258da7cadad6085e78f6fea556ece66bb not found: ID does not exist" Nov 03 21:11:01.893317 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:01.893281 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft" Nov 03 21:11:01.895955 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:01.895933 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9"] Nov 03 21:11:01.946127 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:01.946051 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-w2dr9"] Nov 03 21:11:01.987774 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:01.987738 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/365e3804-37c5-4398-95c0-a1602237453e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/365e3804-37c5-4398-95c0-a1602237453e-istio-envoy\") pod \"365e3804-37c5-4398-95c0-a1602237453e\" (UID: \"365e3804-37c5-4398-95c0-a1602237453e\") " Nov 03 21:11:01.987939 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:01.987865 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/365e3804-37c5-4398-95c0-a1602237453e-istio-token\" (UniqueName: \"kubernetes.io/projected/365e3804-37c5-4398-95c0-a1602237453e-istio-token\") pod \"365e3804-37c5-4398-95c0-a1602237453e\" (UID: \"365e3804-37c5-4398-95c0-a1602237453e\") " Nov 03 21:11:01.987939 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:01.987892 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/365e3804-37c5-4398-95c0-a1602237453e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/365e3804-37c5-4398-95c0-a1602237453e-istiod-ca-cert\") pod \"365e3804-37c5-4398-95c0-a1602237453e\" (UID: \"365e3804-37c5-4398-95c0-a1602237453e\") " Nov 03 21:11:01.987939 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:01.987927 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/365e3804-37c5-4398-95c0-a1602237453e-kube-api-access-c8ps4\" (UniqueName: \"kubernetes.io/projected/365e3804-37c5-4398-95c0-a1602237453e-kube-api-access-c8ps4\") pod \"365e3804-37c5-4398-95c0-a1602237453e\" (UID: \"365e3804-37c5-4398-95c0-a1602237453e\") " Nov 03 21:11:01.988085 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:01.988004 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/365e3804-37c5-4398-95c0-a1602237453e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/365e3804-37c5-4398-95c0-a1602237453e-credential-socket\") pod \"365e3804-37c5-4398-95c0-a1602237453e\" (UID: \"365e3804-37c5-4398-95c0-a1602237453e\") " Nov 03 21:11:01.988085 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:01.988031 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/365e3804-37c5-4398-95c0-a1602237453e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/365e3804-37c5-4398-95c0-a1602237453e-workload-certs\") pod \"365e3804-37c5-4398-95c0-a1602237453e\" (UID: \"365e3804-37c5-4398-95c0-a1602237453e\") " Nov 03 21:11:01.988085 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:01.988068 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/365e3804-37c5-4398-95c0-a1602237453e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/365e3804-37c5-4398-95c0-a1602237453e-istio-data\") pod \"365e3804-37c5-4398-95c0-a1602237453e\" (UID: \"365e3804-37c5-4398-95c0-a1602237453e\") " Nov 03 21:11:01.988321 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:01.988173 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/365e3804-37c5-4398-95c0-a1602237453e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/365e3804-37c5-4398-95c0-a1602237453e-workload-socket\") pod \"365e3804-37c5-4398-95c0-a1602237453e\" (UID: \"365e3804-37c5-4398-95c0-a1602237453e\") " Nov 03 21:11:01.988321 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:01.988209 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/365e3804-37c5-4398-95c0-a1602237453e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/365e3804-37c5-4398-95c0-a1602237453e-istio-podinfo\") pod \"365e3804-37c5-4398-95c0-a1602237453e\" (UID: \"365e3804-37c5-4398-95c0-a1602237453e\") " Nov 03 21:11:01.988481 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:01.988336 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/365e3804-37c5-4398-95c0-a1602237453e-credential-socket" pod "365e3804-37c5-4398-95c0-a1602237453e" (UID: "365e3804-37c5-4398-95c0-a1602237453e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:11:01.988481 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:01.988350 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/365e3804-37c5-4398-95c0-a1602237453e-istiod-ca-cert" pod "365e3804-37c5-4398-95c0-a1602237453e" (UID: "365e3804-37c5-4398-95c0-a1602237453e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:11:01.988634 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:01.988528 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/365e3804-37c5-4398-95c0-a1602237453e-istio-data" pod "365e3804-37c5-4398-95c0-a1602237453e" (UID: "365e3804-37c5-4398-95c0-a1602237453e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:11:01.988823 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:01.988618 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/365e3804-37c5-4398-95c0-a1602237453e-workload-socket" pod "365e3804-37c5-4398-95c0-a1602237453e" (UID: "365e3804-37c5-4398-95c0-a1602237453e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:11:01.989015 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:01.988953 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/365e3804-37c5-4398-95c0-a1602237453e-istiod-ca-cert\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:11:01.989015 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:01.988975 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/365e3804-37c5-4398-95c0-a1602237453e-credential-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:11:01.989015 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:01.988992 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/365e3804-37c5-4398-95c0-a1602237453e-istio-data\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:11:01.989015 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:01.989006 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/365e3804-37c5-4398-95c0-a1602237453e-workload-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:11:01.989237 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:01.989220 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/365e3804-37c5-4398-95c0-a1602237453e-workload-certs" pod "365e3804-37c5-4398-95c0-a1602237453e" (UID: "365e3804-37c5-4398-95c0-a1602237453e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:11:01.990509 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:01.990483 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/365e3804-37c5-4398-95c0-a1602237453e-istio-envoy" pod "365e3804-37c5-4398-95c0-a1602237453e" (UID: "365e3804-37c5-4398-95c0-a1602237453e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:11:01.990654 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:01.990569 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/365e3804-37c5-4398-95c0-a1602237453e-istio-token" pod "365e3804-37c5-4398-95c0-a1602237453e" (UID: "365e3804-37c5-4398-95c0-a1602237453e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:11:01.990654 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:01.990603 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/365e3804-37c5-4398-95c0-a1602237453e-kube-api-access-c8ps4" pod "365e3804-37c5-4398-95c0-a1602237453e" (UID: "365e3804-37c5-4398-95c0-a1602237453e"). InnerVolumeSpecName "kube-api-access-c8ps4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:11:01.991069 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:01.991049 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/365e3804-37c5-4398-95c0-a1602237453e-istio-podinfo" pod "365e3804-37c5-4398-95c0-a1602237453e" (UID: "365e3804-37c5-4398-95c0-a1602237453e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:11:02.090935 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:02.090900 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/365e3804-37c5-4398-95c0-a1602237453e-istio-token\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:11:02.090935 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:02.090933 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8ps4\" (UniqueName: \"kubernetes.io/projected/365e3804-37c5-4398-95c0-a1602237453e-kube-api-access-c8ps4\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:11:02.090935 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:02.090944 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/365e3804-37c5-4398-95c0-a1602237453e-workload-certs\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:11:02.091183 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:02.090953 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/365e3804-37c5-4398-95c0-a1602237453e-istio-podinfo\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:11:02.091183 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:02.090962 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/365e3804-37c5-4398-95c0-a1602237453e-istio-envoy\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:11:02.716780 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:02.716747 4305 generic.go:356] "Generic (PLEG): container finished" podID="365e3804-37c5-4398-95c0-a1602237453e" containerID="097e2b681c7777e9b661d86817933bb1d1b6cbd767c560ff92ea59a4807a981a" exitCode=0 Nov 03 21:11:02.717290 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:02.716803 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft" Nov 03 21:11:02.717290 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:02.716830 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft" event={"ID":"365e3804-37c5-4398-95c0-a1602237453e","Type":"ContainerDied","Data":"097e2b681c7777e9b661d86817933bb1d1b6cbd767c560ff92ea59a4807a981a"} Nov 03 21:11:02.717290 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:02.716888 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft" event={"ID":"365e3804-37c5-4398-95c0-a1602237453e","Type":"ContainerDied","Data":"fa88a7604d023ddbf3be4ca2411a0f05afc49cda9bec12a206ee10f57e94bf67"} Nov 03 21:11:02.717290 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:02.716919 4305 scope.go:122] "RemoveContainer" containerID="097e2b681c7777e9b661d86817933bb1d1b6cbd767c560ff92ea59a4807a981a" Nov 03 21:11:02.728603 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:02.728584 4305 scope.go:122] "RemoveContainer" containerID="8c843261fb8ad8e8ef30b9bc672917ec5843dfc9995c59389d3b8fe006d9a0f2" Nov 03 21:11:02.738823 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:02.738803 4305 scope.go:122] "RemoveContainer" containerID="097e2b681c7777e9b661d86817933bb1d1b6cbd767c560ff92ea59a4807a981a" Nov 03 21:11:02.739158 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:11:02.739134 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"097e2b681c7777e9b661d86817933bb1d1b6cbd767c560ff92ea59a4807a981a\": container with ID starting with 097e2b681c7777e9b661d86817933bb1d1b6cbd767c560ff92ea59a4807a981a not found: ID does not exist" containerID="097e2b681c7777e9b661d86817933bb1d1b6cbd767c560ff92ea59a4807a981a" Nov 03 21:11:02.739213 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:02.739168 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"097e2b681c7777e9b661d86817933bb1d1b6cbd767c560ff92ea59a4807a981a"} err="failed to get container status \"097e2b681c7777e9b661d86817933bb1d1b6cbd767c560ff92ea59a4807a981a\": rpc error: code = NotFound desc = could not find container \"097e2b681c7777e9b661d86817933bb1d1b6cbd767c560ff92ea59a4807a981a\": container with ID starting with 097e2b681c7777e9b661d86817933bb1d1b6cbd767c560ff92ea59a4807a981a not found: ID does not exist" Nov 03 21:11:02.739213 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:02.739188 4305 scope.go:122] "RemoveContainer" containerID="8c843261fb8ad8e8ef30b9bc672917ec5843dfc9995c59389d3b8fe006d9a0f2" Nov 03 21:11:02.739452 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:11:02.739418 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c843261fb8ad8e8ef30b9bc672917ec5843dfc9995c59389d3b8fe006d9a0f2\": container with ID starting with 8c843261fb8ad8e8ef30b9bc672917ec5843dfc9995c59389d3b8fe006d9a0f2 not found: ID does not exist" containerID="8c843261fb8ad8e8ef30b9bc672917ec5843dfc9995c59389d3b8fe006d9a0f2" Nov 03 21:11:02.739518 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:02.739464 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c843261fb8ad8e8ef30b9bc672917ec5843dfc9995c59389d3b8fe006d9a0f2"} err="failed to get container status \"8c843261fb8ad8e8ef30b9bc672917ec5843dfc9995c59389d3b8fe006d9a0f2\": rpc error: code = NotFound desc = could not find container \"8c843261fb8ad8e8ef30b9bc672917ec5843dfc9995c59389d3b8fe006d9a0f2\": container with ID starting with 8c843261fb8ad8e8ef30b9bc672917ec5843dfc9995c59389d3b8fe006d9a0f2 not found: ID does not exist" Nov 03 21:11:02.899078 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:02.899045 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f8328f2-ffba-400c-b4e3-c6c62092fefd" path="/var/lib/kubelet/pods/9f8328f2-ffba-400c-b4e3-c6c62092fefd/volumes" Nov 03 21:11:02.899397 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:02.899382 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft"] Nov 03 21:11:02.938588 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:02.938553 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ca4642-openshift-default-6b89f76569-qbgft"] Nov 03 21:11:04.898026 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:04.897991 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="365e3804-37c5-4398-95c0-a1602237453e" path="/var/lib/kubelet/pods/365e3804-37c5-4398-95c0-a1602237453e/volumes" Nov 03 21:11:11.048630 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:11.048598 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2ec0949-qcjkc"] Nov 03 21:11:11.098187 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:11.098152 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2ec0949-qcjkc"] Nov 03 21:11:11.711088 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:11.711061 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-01c7a3-55bcd6997b-k8qhc" Nov 03 21:11:11.711277 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:11.711127 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01c7a3-6479dcc9-xzqvp" Nov 03 21:11:12.903990 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:12.903948 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc4e2350-e601-418a-a278-05bb539d4cef" path="/var/lib/kubelet/pods/dc4e2350-e601-418a-a278-05bb539d4cef/volumes" Nov 03 21:11:16.323373 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:11:16.323334 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:11:19.306462 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:19.304629 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-206508-9hwdl"] Nov 03 21:11:19.373456 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:19.373118 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-206508-9hwdl"] Nov 03 21:11:20.622609 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:20.622576 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-499187-vqvzq"] Nov 03 21:11:20.671597 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:20.671562 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-499187-vqvzq"] Nov 03 21:11:20.897079 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:20.897041 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="772927b9-7b5b-4dc7-b62e-c5dd61b54960" path="/var/lib/kubelet/pods/772927b9-7b5b-4dc7-b62e-c5dd61b54960/volumes" Nov 03 21:11:20.897581 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:20.897566 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="964b98c5-2a4e-41e8-b707-4f2915e61933" path="/var/lib/kubelet/pods/964b98c5-2a4e-41e8-b707-4f2915e61933/volumes" Nov 03 21:11:26.111220 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:26.111160 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:11:28.480682 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:28.480644 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-218976-9dpnc"] Nov 03 21:11:28.481322 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:28.481308 4305 cpu_manager.go:414] "containerMap: removing container" podUID="365e3804-37c5-4398-95c0-a1602237453e" containerName="istio-proxy" Nov 03 21:11:28.481379 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:28.481324 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="365e3804-37c5-4398-95c0-a1602237453e" containerName="istio-proxy" Nov 03 21:11:28.481379 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:28.481367 4305 cpu_manager.go:414] "containerMap: removing container" podUID="9f8328f2-ffba-400c-b4e3-c6c62092fefd" containerName="istio-proxy" Nov 03 21:11:28.481379 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:28.481372 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f8328f2-ffba-400c-b4e3-c6c62092fefd" containerName="istio-proxy" Nov 03 21:11:28.481514 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:28.481502 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="365e3804-37c5-4398-95c0-a1602237453e" containerName="istio-proxy" Nov 03 21:11:28.481561 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:28.481528 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f8328f2-ffba-400c-b4e3-c6c62092fefd" containerName="istio-proxy" Nov 03 21:11:28.486198 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:28.486180 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-218976-9dpnc" Nov 03 21:11:28.528274 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:28.528242 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-218976-9dpnc"] Nov 03 21:11:28.576596 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:28.576563 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:11:28.586555 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:28.576954 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:11:28.586858 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:28.577071 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77vtd\"" Nov 03 21:11:28.586999 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:28.577143 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:11:28.605341 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:28.605304 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwz8v\" (UniqueName: \"kubernetes.io/projected/c8c93176-b632-4831-b43d-2882a1c2f659-kube-api-access-wwz8v\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-218976-9dpnc\" (UID: \"c8c93176-b632-4831-b43d-2882a1c2f659\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-218976-9dpnc" Nov 03 21:11:28.605604 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:28.605455 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8c93176-b632-4831-b43d-2882a1c2f659-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-218976-9dpnc\" (UID: \"c8c93176-b632-4831-b43d-2882a1c2f659\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-218976-9dpnc" Nov 03 21:11:28.605698 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:28.605631 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8c93176-b632-4831-b43d-2882a1c2f659-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-218976-9dpnc\" (UID: \"c8c93176-b632-4831-b43d-2882a1c2f659\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-218976-9dpnc" Nov 03 21:11:28.707465 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:28.707397 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8c93176-b632-4831-b43d-2882a1c2f659-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-218976-9dpnc\" (UID: \"c8c93176-b632-4831-b43d-2882a1c2f659\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-218976-9dpnc" Nov 03 21:11:28.707650 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:28.707624 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8c93176-b632-4831-b43d-2882a1c2f659-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-218976-9dpnc\" (UID: \"c8c93176-b632-4831-b43d-2882a1c2f659\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-218976-9dpnc" Nov 03 21:11:28.707748 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:28.707731 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwz8v\" (UniqueName: \"kubernetes.io/projected/c8c93176-b632-4831-b43d-2882a1c2f659-kube-api-access-wwz8v\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-218976-9dpnc\" (UID: \"c8c93176-b632-4831-b43d-2882a1c2f659\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-218976-9dpnc" Nov 03 21:11:28.708284 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:28.708260 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8c93176-b632-4831-b43d-2882a1c2f659-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-218976-9dpnc\" (UID: \"c8c93176-b632-4831-b43d-2882a1c2f659\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-218976-9dpnc" Nov 03 21:11:28.710016 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:28.709997 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8c93176-b632-4831-b43d-2882a1c2f659-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-218976-9dpnc\" (UID: \"c8c93176-b632-4831-b43d-2882a1c2f659\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-218976-9dpnc" Nov 03 21:11:28.780506 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:28.780235 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwz8v\" (UniqueName: \"kubernetes.io/projected/c8c93176-b632-4831-b43d-2882a1c2f659-kube-api-access-wwz8v\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-218976-9dpnc\" (UID: \"c8c93176-b632-4831-b43d-2882a1c2f659\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-218976-9dpnc" Nov 03 21:11:28.800780 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:28.800748 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-218976-9dpnc" Nov 03 21:11:29.343541 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:29.334044 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-218976-9dpnc"] Nov 03 21:11:29.348938 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:11:29.348891 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8c93176_b632_4831_b43d_2882a1c2f659.slice/crio-2e2fe51b3101108bf7fd09512fbf1ff18132582384329c0c287580db34795315 WatchSource:0}: Error finding container 2e2fe51b3101108bf7fd09512fbf1ff18132582384329c0c287580db34795315: Status 404 returned error can't find the container with id 2e2fe51b3101108bf7fd09512fbf1ff18132582384329c0c287580db34795315 Nov 03 21:11:29.934971 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:29.934931 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-218976-9dpnc" event={"ID":"c8c93176-b632-4831-b43d-2882a1c2f659","Type":"ContainerStarted","Data":"e8d5b09aeadb497765f2889e8a7c6a7abfe40a8d6dab0921340fd82d1a3f8202"} Nov 03 21:11:29.935514 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:29.934995 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-218976-9dpnc" event={"ID":"c8c93176-b632-4831-b43d-2882a1c2f659","Type":"ContainerStarted","Data":"2e2fe51b3101108bf7fd09512fbf1ff18132582384329c0c287580db34795315"} Nov 03 21:11:30.079374 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:30.079298 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-218976-9dpnc" podStartSLOduration=2.079275426 podStartE2EDuration="2.079275426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:11:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:11:30.076013182 +0000 UTC m=+1925.301332997" watchObservedRunningTime="2026-11-03 21:11:30.079275426 +0000 UTC m=+1925.304595220" Nov 03 21:11:30.944189 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:30.944149 4305 generic.go:356] "Generic (PLEG): container finished" podID="c8c93176-b632-4831-b43d-2882a1c2f659" containerID="e8d5b09aeadb497765f2889e8a7c6a7abfe40a8d6dab0921340fd82d1a3f8202" exitCode=0 Nov 03 21:11:30.944620 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:30.944256 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-218976-9dpnc" event={"ID":"c8c93176-b632-4831-b43d-2882a1c2f659","Type":"ContainerDied","Data":"e8d5b09aeadb497765f2889e8a7c6a7abfe40a8d6dab0921340fd82d1a3f8202"} Nov 03 21:11:31.678899 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:31.678858 4305 scope.go:122] "RemoveContainer" containerID="414f3f706b4a19d584b1da213d1240e611c35fb834c30a970397ed3e26fe038d" Nov 03 21:11:31.689872 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:31.689845 4305 scope.go:122] "RemoveContainer" containerID="9d1a3c0e8fa874ede2e654b37c318cc124a719cc769bb43040b87736e8603472" Nov 03 21:11:32.148959 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:32.148929 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-218976-9dpnc" Nov 03 21:11:32.206011 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:32.205936 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8c93176-b632-4831-b43d-2882a1c2f659-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8c93176-b632-4831-b43d-2882a1c2f659-maas-api-xks-ca\") pod \"c8c93176-b632-4831-b43d-2882a1c2f659\" (UID: \"c8c93176-b632-4831-b43d-2882a1c2f659\") " Nov 03 21:11:32.206174 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:32.206013 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8c93176-b632-4831-b43d-2882a1c2f659-kube-api-access-wwz8v\" (UniqueName: \"kubernetes.io/projected/c8c93176-b632-4831-b43d-2882a1c2f659-kube-api-access-wwz8v\") pod \"c8c93176-b632-4831-b43d-2882a1c2f659\" (UID: \"c8c93176-b632-4831-b43d-2882a1c2f659\") " Nov 03 21:11:32.206174 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:32.206159 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8c93176-b632-4831-b43d-2882a1c2f659-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8c93176-b632-4831-b43d-2882a1c2f659-maas-api-openshift-ca\") pod \"c8c93176-b632-4831-b43d-2882a1c2f659\" (UID: \"c8c93176-b632-4831-b43d-2882a1c2f659\") " Nov 03 21:11:32.206545 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:32.206513 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8c93176-b632-4831-b43d-2882a1c2f659-maas-api-openshift-ca" pod "c8c93176-b632-4831-b43d-2882a1c2f659" (UID: "c8c93176-b632-4831-b43d-2882a1c2f659"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:11:32.208147 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:32.208121 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8c93176-b632-4831-b43d-2882a1c2f659-maas-api-xks-ca" pod "c8c93176-b632-4831-b43d-2882a1c2f659" (UID: "c8c93176-b632-4831-b43d-2882a1c2f659"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:11:32.208299 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:32.208184 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8c93176-b632-4831-b43d-2882a1c2f659-kube-api-access-wwz8v" pod "c8c93176-b632-4831-b43d-2882a1c2f659" (UID: "c8c93176-b632-4831-b43d-2882a1c2f659"). InnerVolumeSpecName "kube-api-access-wwz8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:11:32.307758 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:32.307721 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8c93176-b632-4831-b43d-2882a1c2f659-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:11:32.307758 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:32.307757 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8c93176-b632-4831-b43d-2882a1c2f659-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:11:32.307758 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:32.307767 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwz8v\" (UniqueName: \"kubernetes.io/projected/c8c93176-b632-4831-b43d-2882a1c2f659-kube-api-access-wwz8v\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:11:32.962520 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:32.962490 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-218976-9dpnc" Nov 03 21:11:32.962520 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:32.962500 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-218976-9dpnc" event={"ID":"c8c93176-b632-4831-b43d-2882a1c2f659","Type":"ContainerDied","Data":"2e2fe51b3101108bf7fd09512fbf1ff18132582384329c0c287580db34795315"} Nov 03 21:11:32.962748 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:32.962547 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e2fe51b3101108bf7fd09512fbf1ff18132582384329c0c287580db34795315" Nov 03 21:11:36.996165 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:36.996126 4305 generic.go:356] "Generic (PLEG): container finished" podID="953874ed-6376-4775-b514-b09ecb325824" containerID="ce43d975478cc8a99b640aee6c05a90a6cf327d4f1d6852ae96596c44910f64d" exitCode=0 Nov 03 21:11:36.996647 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:36.996193 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr" event={"ID":"953874ed-6376-4775-b514-b09ecb325824","Type":"ContainerDied","Data":"ce43d975478cc8a99b640aee6c05a90a6cf327d4f1d6852ae96596c44910f64d"} Nov 03 21:11:36.996696 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:36.996677 4305 scope.go:122] "RemoveContainer" containerID="ce43d975478cc8a99b640aee6c05a90a6cf327d4f1d6852ae96596c44910f64d" Nov 03 21:11:36.998516 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:36.998482 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:11:36.998653 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:36.998575 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:11:36.998720 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:36.998650 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:11:38.006525 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:38.006493 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr" event={"ID":"953874ed-6376-4775-b514-b09ecb325824","Type":"ContainerStarted","Data":"3e9b081e86ee58626087431ea3a50f9d3b1d0fdd64224897f303ba82806b26ff"} Nov 03 21:11:38.020040 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:38.019996 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr" Nov 03 21:11:38.020291 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:38.020266 4305 patch_prober.go:29] interesting pod/e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Nov 03 21:11:38.020406 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:38.020339 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr" podUID="953874ed-6376-4775-b514-b09ecb325824" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Nov 03 21:11:39.019694 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:39.019655 4305 patch_prober.go:29] interesting pod/e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Nov 03 21:11:39.020241 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:39.019712 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr" podUID="953874ed-6376-4775-b514-b09ecb325824" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Nov 03 21:11:40.019422 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:40.019388 4305 patch_prober.go:29] interesting pod/e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Nov 03 21:11:40.019646 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:40.019460 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr" podUID="953874ed-6376-4775-b514-b09ecb325824" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Nov 03 21:11:41.020507 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:41.020466 4305 patch_prober.go:29] interesting pod/e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Nov 03 21:11:41.020900 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:41.020521 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr" podUID="953874ed-6376-4775-b514-b09ecb325824" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Nov 03 21:11:42.024688 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:42.024633 4305 patch_prober.go:29] interesting pod/e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Nov 03 21:11:42.025366 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:42.025331 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr" podUID="953874ed-6376-4775-b514-b09ecb325824" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Nov 03 21:11:43.019898 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:43.019865 4305 patch_prober.go:29] interesting pod/e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Nov 03 21:11:43.020096 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:43.019942 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr" podUID="953874ed-6376-4775-b514-b09ecb325824" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Nov 03 21:11:44.025294 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:44.025261 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr" Nov 03 21:11:44.025720 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:44.025693 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr" Nov 03 21:11:44.026382 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:44.026362 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr" Nov 03 21:11:46.322465 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:11:46.322396 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:11:49.494511 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:11:49.494477 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:12:00.200871 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:00.200836 4305 generic.go:356] "Generic (PLEG): container finished" podID="e4063115-7846-42ea-8bea-4684c27b132a" containerID="18c8bfc24d304e2064190791989c8695d18a0eaf3e807aef35efaa067e6aa293" exitCode=0 Nov 03 21:12:00.201346 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:00.200913 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs" event={"ID":"e4063115-7846-42ea-8bea-4684c27b132a","Type":"ContainerDied","Data":"18c8bfc24d304e2064190791989c8695d18a0eaf3e807aef35efaa067e6aa293"} Nov 03 21:12:00.201393 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:00.201363 4305 scope.go:122] "RemoveContainer" containerID="18c8bfc24d304e2064190791989c8695d18a0eaf3e807aef35efaa067e6aa293" Nov 03 21:12:00.203050 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:00.203010 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:12:00.203125 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:00.203093 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:12:00.203161 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:00.203128 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:12:01.210032 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:01.209994 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs" event={"ID":"e4063115-7846-42ea-8bea-4684c27b132a","Type":"ContainerStarted","Data":"9795139d2fdc6878b2ca6a18c3b3908b403cd54e700ce49c53546f48b6ace684"} Nov 03 21:12:02.141825 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:02.141786 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs" Nov 03 21:12:02.142021 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:02.141997 4305 patch_prober.go:29] interesting pod/e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Nov 03 21:12:02.142081 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:02.142060 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs" podUID="e4063115-7846-42ea-8bea-4684c27b132a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Nov 03 21:12:03.141983 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:03.141941 4305 patch_prober.go:29] interesting pod/e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Nov 03 21:12:03.142451 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:03.142030 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs" podUID="e4063115-7846-42ea-8bea-4684c27b132a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Nov 03 21:12:04.142543 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:04.142505 4305 patch_prober.go:29] interesting pod/e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Nov 03 21:12:04.142970 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:04.142592 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs" podUID="e4063115-7846-42ea-8bea-4684c27b132a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Nov 03 21:12:05.142180 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:05.142142 4305 patch_prober.go:29] interesting pod/e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Nov 03 21:12:05.142372 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:05.142199 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs" podUID="e4063115-7846-42ea-8bea-4684c27b132a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Nov 03 21:12:06.141883 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:06.141835 4305 patch_prober.go:29] interesting pod/e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Nov 03 21:12:06.143249 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:06.143216 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs" podUID="e4063115-7846-42ea-8bea-4684c27b132a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Nov 03 21:12:07.147015 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:07.146982 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs" Nov 03 21:12:07.147443 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:07.147398 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs" Nov 03 21:12:07.148288 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:07.148264 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs" Nov 03 21:12:16.321954 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:12:16.321905 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:12:18.416961 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:18.416932 4305 generic.go:356] "Generic (PLEG): container finished" podID="239c6c9d-3ac5-49b5-9093-e686f002ce2a" containerID="0eeba93b3f6bc986d8eda7e3d9427349d5e1c7704233e90443cdd1a54dd67d1d" exitCode=0 Nov 03 21:12:18.417452 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:18.417015 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v" event={"ID":"239c6c9d-3ac5-49b5-9093-e686f002ce2a","Type":"ContainerDied","Data":"0eeba93b3f6bc986d8eda7e3d9427349d5e1c7704233e90443cdd1a54dd67d1d"} Nov 03 21:12:18.417512 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:18.417456 4305 scope.go:122] "RemoveContainer" containerID="0eeba93b3f6bc986d8eda7e3d9427349d5e1c7704233e90443cdd1a54dd67d1d" Nov 03 21:12:18.418991 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:18.418964 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:12:18.419056 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:18.419035 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:12:18.419095 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:18.419072 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:12:19.428946 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:19.428910 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v" event={"ID":"239c6c9d-3ac5-49b5-9093-e686f002ce2a","Type":"ContainerStarted","Data":"12c5a83a46c15be46fbf63f95d841479f29d7292d53fb8370968442f3eb2ea31"} Nov 03 21:12:20.106950 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:20.106820 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v" Nov 03 21:12:20.108366 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:20.108000 4305 patch_prober.go:29] interesting pod/e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Nov 03 21:12:20.108366 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:20.108074 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v" podUID="239c6c9d-3ac5-49b5-9093-e686f002ce2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Nov 03 21:12:20.531048 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:20.530990 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v" Nov 03 21:12:21.103174 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:21.103138 4305 patch_prober.go:29] interesting pod/e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Nov 03 21:12:21.103372 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:21.103200 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v" podUID="239c6c9d-3ac5-49b5-9093-e686f002ce2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Nov 03 21:12:22.103039 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:22.103000 4305 patch_prober.go:29] interesting pod/e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Nov 03 21:12:22.103478 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:22.103075 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v" podUID="239c6c9d-3ac5-49b5-9093-e686f002ce2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Nov 03 21:12:23.103577 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:23.103538 4305 patch_prober.go:29] interesting pod/e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Nov 03 21:12:23.104020 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:23.103595 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v" podUID="239c6c9d-3ac5-49b5-9093-e686f002ce2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Nov 03 21:12:24.103857 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:24.103822 4305 patch_prober.go:29] interesting pod/e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Nov 03 21:12:24.104330 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:24.103882 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v" podUID="239c6c9d-3ac5-49b5-9093-e686f002ce2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Nov 03 21:12:25.108514 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:25.108479 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v" Nov 03 21:12:25.109567 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:25.109545 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v" Nov 03 21:12:27.030908 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:27.030866 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl"] Nov 03 21:12:27.511072 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:27.511013 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl" podUID="90f310bb-3fa1-404e-b893-6c49a2f760e1" containerName="istio-proxy" containerID="cri-o://b47203e5f72b4728f861c46b62dec7d58a6a684008e7cff760d6c95f8f20cde0" gracePeriod=30 Nov 03 21:12:28.928025 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:28.927988 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:12:31.162527 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:31.162488 4305 patch_prober.go:29] interesting pod/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:12:31.163027 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:31.162564 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl" podUID="90f310bb-3fa1-404e-b893-6c49a2f760e1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:12:31.744167 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:31.744134 4305 scope.go:122] "RemoveContainer" containerID="d3aa7304321297086ece0fc9826a71c049fc17a184f92b2c57efbe3ab84d6a0c" Nov 03 21:12:31.756993 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:31.756968 4305 scope.go:122] "RemoveContainer" containerID="1304695d1bf953312434c531d0ac50ecf013ac5c92257b4d2358e7afff19fd6f" Nov 03 21:12:32.034081 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:32.033998 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-218976-9dpnc"] Nov 03 21:12:32.082667 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:32.082633 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-218976-9dpnc"] Nov 03 21:12:32.784465 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:32.784384 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl" Nov 03 21:12:32.874841 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:32.874801 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/90f310bb-3fa1-404e-b893-6c49a2f760e1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90f310bb-3fa1-404e-b893-6c49a2f760e1-istio-podinfo\") pod \"90f310bb-3fa1-404e-b893-6c49a2f760e1\" (UID: \"90f310bb-3fa1-404e-b893-6c49a2f760e1\") " Nov 03 21:12:32.875051 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:32.874892 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90f310bb-3fa1-404e-b893-6c49a2f760e1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f310bb-3fa1-404e-b893-6c49a2f760e1-workload-socket\") pod \"90f310bb-3fa1-404e-b893-6c49a2f760e1\" (UID: \"90f310bb-3fa1-404e-b893-6c49a2f760e1\") " Nov 03 21:12:32.875051 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:32.874916 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90f310bb-3fa1-404e-b893-6c49a2f760e1-istio-token\" (UniqueName: \"kubernetes.io/projected/90f310bb-3fa1-404e-b893-6c49a2f760e1-istio-token\") pod \"90f310bb-3fa1-404e-b893-6c49a2f760e1\" (UID: \"90f310bb-3fa1-404e-b893-6c49a2f760e1\") " Nov 03 21:12:32.875051 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:32.874962 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90f310bb-3fa1-404e-b893-6c49a2f760e1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90f310bb-3fa1-404e-b893-6c49a2f760e1-istio-envoy\") pod \"90f310bb-3fa1-404e-b893-6c49a2f760e1\" (UID: \"90f310bb-3fa1-404e-b893-6c49a2f760e1\") " Nov 03 21:12:32.875051 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:32.874993 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90f310bb-3fa1-404e-b893-6c49a2f760e1-kube-api-access-g8gxx\" (UniqueName: \"kubernetes.io/projected/90f310bb-3fa1-404e-b893-6c49a2f760e1-kube-api-access-g8gxx\") pod \"90f310bb-3fa1-404e-b893-6c49a2f760e1\" (UID: \"90f310bb-3fa1-404e-b893-6c49a2f760e1\") " Nov 03 21:12:32.875289 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:32.875154 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90f310bb-3fa1-404e-b893-6c49a2f760e1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90f310bb-3fa1-404e-b893-6c49a2f760e1-workload-certs\") pod \"90f310bb-3fa1-404e-b893-6c49a2f760e1\" (UID: \"90f310bb-3fa1-404e-b893-6c49a2f760e1\") " Nov 03 21:12:32.875289 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:32.875216 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90f310bb-3fa1-404e-b893-6c49a2f760e1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90f310bb-3fa1-404e-b893-6c49a2f760e1-istio-data\") pod \"90f310bb-3fa1-404e-b893-6c49a2f760e1\" (UID: \"90f310bb-3fa1-404e-b893-6c49a2f760e1\") " Nov 03 21:12:32.875289 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:32.875248 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90f310bb-3fa1-404e-b893-6c49a2f760e1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f310bb-3fa1-404e-b893-6c49a2f760e1-credential-socket\") pod \"90f310bb-3fa1-404e-b893-6c49a2f760e1\" (UID: \"90f310bb-3fa1-404e-b893-6c49a2f760e1\") " Nov 03 21:12:32.875469 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:32.875335 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90f310bb-3fa1-404e-b893-6c49a2f760e1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90f310bb-3fa1-404e-b893-6c49a2f760e1-istiod-ca-cert\") pod \"90f310bb-3fa1-404e-b893-6c49a2f760e1\" (UID: \"90f310bb-3fa1-404e-b893-6c49a2f760e1\") " Nov 03 21:12:32.875469 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:32.875211 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90f310bb-3fa1-404e-b893-6c49a2f760e1-workload-socket" pod "90f310bb-3fa1-404e-b893-6c49a2f760e1" (UID: "90f310bb-3fa1-404e-b893-6c49a2f760e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:12:32.875469 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:32.875362 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90f310bb-3fa1-404e-b893-6c49a2f760e1-workload-certs" pod "90f310bb-3fa1-404e-b893-6c49a2f760e1" (UID: "90f310bb-3fa1-404e-b893-6c49a2f760e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:12:32.875650 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:32.875526 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90f310bb-3fa1-404e-b893-6c49a2f760e1-istio-data" pod "90f310bb-3fa1-404e-b893-6c49a2f760e1" (UID: "90f310bb-3fa1-404e-b893-6c49a2f760e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:12:32.875866 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:32.875841 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90f310bb-3fa1-404e-b893-6c49a2f760e1-istiod-ca-cert" pod "90f310bb-3fa1-404e-b893-6c49a2f760e1" (UID: "90f310bb-3fa1-404e-b893-6c49a2f760e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:12:32.875950 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:32.875900 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90f310bb-3fa1-404e-b893-6c49a2f760e1-credential-socket" pod "90f310bb-3fa1-404e-b893-6c49a2f760e1" (UID: "90f310bb-3fa1-404e-b893-6c49a2f760e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:12:32.876031 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:32.876013 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f310bb-3fa1-404e-b893-6c49a2f760e1-workload-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:12:32.876078 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:32.876040 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90f310bb-3fa1-404e-b893-6c49a2f760e1-workload-certs\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:12:32.876078 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:32.876054 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90f310bb-3fa1-404e-b893-6c49a2f760e1-istio-data\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:12:32.876078 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:32.876067 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f310bb-3fa1-404e-b893-6c49a2f760e1-credential-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:12:32.876238 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:32.876080 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90f310bb-3fa1-404e-b893-6c49a2f760e1-istiod-ca-cert\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:12:32.877485 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:32.877452 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90f310bb-3fa1-404e-b893-6c49a2f760e1-istio-envoy" pod "90f310bb-3fa1-404e-b893-6c49a2f760e1" (UID: "90f310bb-3fa1-404e-b893-6c49a2f760e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:12:32.877893 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:32.877874 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90f310bb-3fa1-404e-b893-6c49a2f760e1-istio-token" pod "90f310bb-3fa1-404e-b893-6c49a2f760e1" (UID: "90f310bb-3fa1-404e-b893-6c49a2f760e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:12:32.877971 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:32.877890 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90f310bb-3fa1-404e-b893-6c49a2f760e1-istio-podinfo" pod "90f310bb-3fa1-404e-b893-6c49a2f760e1" (UID: "90f310bb-3fa1-404e-b893-6c49a2f760e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:12:32.878015 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:32.877976 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90f310bb-3fa1-404e-b893-6c49a2f760e1-kube-api-access-g8gxx" pod "90f310bb-3fa1-404e-b893-6c49a2f760e1" (UID: "90f310bb-3fa1-404e-b893-6c49a2f760e1"). InnerVolumeSpecName "kube-api-access-g8gxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:12:32.896905 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:32.896875 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8c93176-b632-4831-b43d-2882a1c2f659" path="/var/lib/kubelet/pods/c8c93176-b632-4831-b43d-2882a1c2f659/volumes" Nov 03 21:12:32.977420 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:32.977378 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90f310bb-3fa1-404e-b893-6c49a2f760e1-istio-podinfo\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:12:32.977420 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:32.977413 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90f310bb-3fa1-404e-b893-6c49a2f760e1-istio-token\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:12:32.977420 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:32.977426 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90f310bb-3fa1-404e-b893-6c49a2f760e1-istio-envoy\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:12:32.977710 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:32.977482 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8gxx\" (UniqueName: \"kubernetes.io/projected/90f310bb-3fa1-404e-b893-6c49a2f760e1-kube-api-access-g8gxx\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:12:33.566975 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:33.566942 4305 generic.go:356] "Generic (PLEG): container finished" podID="90f310bb-3fa1-404e-b893-6c49a2f760e1" containerID="b47203e5f72b4728f861c46b62dec7d58a6a684008e7cff760d6c95f8f20cde0" exitCode=0 Nov 03 21:12:33.567243 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:33.566991 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl" Nov 03 21:12:33.567243 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:33.567009 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl" event={"ID":"90f310bb-3fa1-404e-b893-6c49a2f760e1","Type":"ContainerDied","Data":"b47203e5f72b4728f861c46b62dec7d58a6a684008e7cff760d6c95f8f20cde0"} Nov 03 21:12:33.567243 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:33.567073 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl" event={"ID":"90f310bb-3fa1-404e-b893-6c49a2f760e1","Type":"ContainerDied","Data":"14eb79710a42b2d5c289648361d3463058e95fd54c76c2aca7dda7d960d17af2"} Nov 03 21:12:33.567243 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:33.567104 4305 scope.go:122] "RemoveContainer" containerID="b47203e5f72b4728f861c46b62dec7d58a6a684008e7cff760d6c95f8f20cde0" Nov 03 21:12:33.578801 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:33.578779 4305 scope.go:122] "RemoveContainer" containerID="b47203e5f72b4728f861c46b62dec7d58a6a684008e7cff760d6c95f8f20cde0" Nov 03 21:12:33.579089 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:12:33.579071 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b47203e5f72b4728f861c46b62dec7d58a6a684008e7cff760d6c95f8f20cde0\": container with ID starting with b47203e5f72b4728f861c46b62dec7d58a6a684008e7cff760d6c95f8f20cde0 not found: ID does not exist" containerID="b47203e5f72b4728f861c46b62dec7d58a6a684008e7cff760d6c95f8f20cde0" Nov 03 21:12:33.579165 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:33.579099 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b47203e5f72b4728f861c46b62dec7d58a6a684008e7cff760d6c95f8f20cde0"} err="failed to get container status \"b47203e5f72b4728f861c46b62dec7d58a6a684008e7cff760d6c95f8f20cde0\": rpc error: code = NotFound desc = could not find container \"b47203e5f72b4728f861c46b62dec7d58a6a684008e7cff760d6c95f8f20cde0\": container with ID starting with b47203e5f72b4728f861c46b62dec7d58a6a684008e7cff760d6c95f8f20cde0 not found: ID does not exist" Nov 03 21:12:33.784374 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:33.784331 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl"] Nov 03 21:12:33.833692 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:33.833612 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-8psrl"] Nov 03 21:12:34.896056 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:34.896021 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90f310bb-3fa1-404e-b893-6c49a2f760e1" path="/var/lib/kubelet/pods/90f310bb-3fa1-404e-b893-6c49a2f760e1/volumes" Nov 03 21:12:36.330687 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:36.330655 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-70fe43-4d9xj"] Nov 03 21:12:36.331275 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:36.331262 4305 cpu_manager.go:414] "containerMap: removing container" podUID="c8c93176-b632-4831-b43d-2882a1c2f659" containerName="revoke-keys" Nov 03 21:12:36.331333 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:36.331276 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8c93176-b632-4831-b43d-2882a1c2f659" containerName="revoke-keys" Nov 03 21:12:36.331333 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:36.331294 4305 cpu_manager.go:414] "containerMap: removing container" podUID="365e3804-37c5-4398-95c0-a1602237453e" containerName="istio-proxy" Nov 03 21:12:36.331333 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:36.331299 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="365e3804-37c5-4398-95c0-a1602237453e" containerName="istio-proxy" Nov 03 21:12:36.331333 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:36.331329 4305 cpu_manager.go:414] "containerMap: removing container" podUID="90f310bb-3fa1-404e-b893-6c49a2f760e1" containerName="istio-proxy" Nov 03 21:12:36.331333 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:36.331334 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90f310bb-3fa1-404e-b893-6c49a2f760e1" containerName="istio-proxy" Nov 03 21:12:36.331526 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:36.331452 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="90f310bb-3fa1-404e-b893-6c49a2f760e1" containerName="istio-proxy" Nov 03 21:12:36.331526 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:36.331467 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8c93176-b632-4831-b43d-2882a1c2f659" containerName="revoke-keys" Nov 03 21:12:36.331526 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:36.331474 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="365e3804-37c5-4398-95c0-a1602237453e" containerName="istio-proxy" Nov 03 21:12:36.334703 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:36.334684 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-70fe43-4d9xj" Nov 03 21:12:36.420225 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:36.420190 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ed41942-1aee-4af5-aff7-96f823fcebff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-70fe43-4d9xj\" (UID: \"4ed41942-1aee-4af5-aff7-96f823fcebff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-70fe43-4d9xj" Nov 03 21:12:36.420410 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:36.420252 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ed41942-1aee-4af5-aff7-96f823fcebff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-70fe43-4d9xj\" (UID: \"4ed41942-1aee-4af5-aff7-96f823fcebff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-70fe43-4d9xj" Nov 03 21:12:36.420531 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:36.420494 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mjxb\" (UniqueName: \"kubernetes.io/projected/4ed41942-1aee-4af5-aff7-96f823fcebff-kube-api-access-9mjxb\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-70fe43-4d9xj\" (UID: \"4ed41942-1aee-4af5-aff7-96f823fcebff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-70fe43-4d9xj" Nov 03 21:12:36.437289 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:36.437261 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:12:36.437467 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:36.437299 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77vtd\"" Nov 03 21:12:36.441546 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:36.441522 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-70fe43-4d9xj"] Nov 03 21:12:36.446323 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:36.446304 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:12:36.446445 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:36.446368 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:12:36.521667 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:36.521626 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ed41942-1aee-4af5-aff7-96f823fcebff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-70fe43-4d9xj\" (UID: \"4ed41942-1aee-4af5-aff7-96f823fcebff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-70fe43-4d9xj" Nov 03 21:12:36.521872 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:36.521699 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ed41942-1aee-4af5-aff7-96f823fcebff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-70fe43-4d9xj\" (UID: \"4ed41942-1aee-4af5-aff7-96f823fcebff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-70fe43-4d9xj" Nov 03 21:12:36.521872 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:36.521812 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mjxb\" (UniqueName: \"kubernetes.io/projected/4ed41942-1aee-4af5-aff7-96f823fcebff-kube-api-access-9mjxb\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-70fe43-4d9xj\" (UID: \"4ed41942-1aee-4af5-aff7-96f823fcebff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-70fe43-4d9xj" Nov 03 21:12:36.522397 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:36.522372 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ed41942-1aee-4af5-aff7-96f823fcebff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-70fe43-4d9xj\" (UID: \"4ed41942-1aee-4af5-aff7-96f823fcebff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-70fe43-4d9xj" Nov 03 21:12:36.524574 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:36.524521 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ed41942-1aee-4af5-aff7-96f823fcebff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-70fe43-4d9xj\" (UID: \"4ed41942-1aee-4af5-aff7-96f823fcebff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-70fe43-4d9xj" Nov 03 21:12:36.589333 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:36.589258 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mjxb\" (UniqueName: \"kubernetes.io/projected/4ed41942-1aee-4af5-aff7-96f823fcebff-kube-api-access-9mjxb\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-70fe43-4d9xj\" (UID: \"4ed41942-1aee-4af5-aff7-96f823fcebff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-70fe43-4d9xj" Nov 03 21:12:36.648203 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:36.648175 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-70fe43-4d9xj" Nov 03 21:12:37.029257 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:37.029231 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-70fe43-4d9xj"] Nov 03 21:12:37.030337 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:12:37.030313 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ed41942_1aee_4af5_aff7_96f823fcebff.slice/crio-90cbdd8e12d212f313af4b2ff4c92770fad8d041e5bb6ab55243ba89146c1584 WatchSource:0}: Error finding container 90cbdd8e12d212f313af4b2ff4c92770fad8d041e5bb6ab55243ba89146c1584: Status 404 returned error can't find the container with id 90cbdd8e12d212f313af4b2ff4c92770fad8d041e5bb6ab55243ba89146c1584 Nov 03 21:12:37.599508 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:37.599428 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-70fe43-4d9xj" event={"ID":"4ed41942-1aee-4af5-aff7-96f823fcebff","Type":"ContainerStarted","Data":"56f8da079380bf114103cfbb00a7a3d91a2e4c69c5a822447d9d03b986dc4712"} Nov 03 21:12:37.599989 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:37.599541 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-70fe43-4d9xj" event={"ID":"4ed41942-1aee-4af5-aff7-96f823fcebff","Type":"ContainerStarted","Data":"90cbdd8e12d212f313af4b2ff4c92770fad8d041e5bb6ab55243ba89146c1584"} Nov 03 21:12:37.719043 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:37.718987 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-70fe43-4d9xj" podStartSLOduration=1.718971477 podStartE2EDuration="1.718971477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:12:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:12:37.714986588 +0000 UTC m=+1992.940306403" watchObservedRunningTime="2026-11-03 21:12:37.718971477 +0000 UTC m=+1992.944291267" Nov 03 21:12:38.183127 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:38.183090 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-2f36ee-zt8xb"] Nov 03 21:12:38.186303 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:38.186284 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-2f36ee-zt8xb" Nov 03 21:12:38.246082 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:38.246044 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-2f36ee-zt8xb"] Nov 03 21:12:38.247010 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:38.246974 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba7962e8-e17e-4579-bdb8-5b26fa9fb39f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-2f36ee-zt8xb\" (UID: \"ba7962e8-e17e-4579-bdb8-5b26fa9fb39f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-2f36ee-zt8xb" Nov 03 21:12:38.247215 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:38.247176 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba7962e8-e17e-4579-bdb8-5b26fa9fb39f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-2f36ee-zt8xb\" (UID: \"ba7962e8-e17e-4579-bdb8-5b26fa9fb39f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-2f36ee-zt8xb" Nov 03 21:12:38.247215 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:38.247211 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8bsj\" (UniqueName: \"kubernetes.io/projected/ba7962e8-e17e-4579-bdb8-5b26fa9fb39f-kube-api-access-c8bsj\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-2f36ee-zt8xb\" (UID: \"ba7962e8-e17e-4579-bdb8-5b26fa9fb39f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-2f36ee-zt8xb" Nov 03 21:12:38.349096 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:38.349008 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba7962e8-e17e-4579-bdb8-5b26fa9fb39f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-2f36ee-zt8xb\" (UID: \"ba7962e8-e17e-4579-bdb8-5b26fa9fb39f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-2f36ee-zt8xb" Nov 03 21:12:38.349267 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:38.349139 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba7962e8-e17e-4579-bdb8-5b26fa9fb39f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-2f36ee-zt8xb\" (UID: \"ba7962e8-e17e-4579-bdb8-5b26fa9fb39f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-2f36ee-zt8xb" Nov 03 21:12:38.349267 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:38.349164 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8bsj\" (UniqueName: \"kubernetes.io/projected/ba7962e8-e17e-4579-bdb8-5b26fa9fb39f-kube-api-access-c8bsj\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-2f36ee-zt8xb\" (UID: \"ba7962e8-e17e-4579-bdb8-5b26fa9fb39f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-2f36ee-zt8xb" Nov 03 21:12:38.349758 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:38.349735 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba7962e8-e17e-4579-bdb8-5b26fa9fb39f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-2f36ee-zt8xb\" (UID: \"ba7962e8-e17e-4579-bdb8-5b26fa9fb39f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-2f36ee-zt8xb" Nov 03 21:12:38.351727 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:38.351704 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba7962e8-e17e-4579-bdb8-5b26fa9fb39f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-2f36ee-zt8xb\" (UID: \"ba7962e8-e17e-4579-bdb8-5b26fa9fb39f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-2f36ee-zt8xb" Nov 03 21:12:38.410183 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:38.410151 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8bsj\" (UniqueName: \"kubernetes.io/projected/ba7962e8-e17e-4579-bdb8-5b26fa9fb39f-kube-api-access-c8bsj\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-2f36ee-zt8xb\" (UID: \"ba7962e8-e17e-4579-bdb8-5b26fa9fb39f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-2f36ee-zt8xb" Nov 03 21:12:38.501889 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:38.501852 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-2f36ee-zt8xb" Nov 03 21:12:38.611137 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:38.610997 4305 generic.go:356] "Generic (PLEG): container finished" podID="4ed41942-1aee-4af5-aff7-96f823fcebff" containerID="56f8da079380bf114103cfbb00a7a3d91a2e4c69c5a822447d9d03b986dc4712" exitCode=0 Nov 03 21:12:38.611137 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:38.611118 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-70fe43-4d9xj" event={"ID":"4ed41942-1aee-4af5-aff7-96f823fcebff","Type":"ContainerDied","Data":"56f8da079380bf114103cfbb00a7a3d91a2e4c69c5a822447d9d03b986dc4712"} Nov 03 21:12:38.886410 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:38.886384 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-2f36ee-zt8xb"] Nov 03 21:12:38.887463 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:12:38.887421 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba7962e8_e17e_4579_bdb8_5b26fa9fb39f.slice/crio-99e7e747ed35f54403f103b8de7844206c0f87552d02a0f4dd999cef7b723cd7 WatchSource:0}: Error finding container 99e7e747ed35f54403f103b8de7844206c0f87552d02a0f4dd999cef7b723cd7: Status 404 returned error can't find the container with id 99e7e747ed35f54403f103b8de7844206c0f87552d02a0f4dd999cef7b723cd7 Nov 03 21:12:39.620288 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:39.620187 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-2f36ee-zt8xb" event={"ID":"ba7962e8-e17e-4579-bdb8-5b26fa9fb39f","Type":"ContainerStarted","Data":"4e25d72425f9f2dd7535c1b42e421decd8efe591593189de2b7ff301599a94f7"} Nov 03 21:12:39.620288 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:39.620253 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-2f36ee-zt8xb" event={"ID":"ba7962e8-e17e-4579-bdb8-5b26fa9fb39f","Type":"ContainerStarted","Data":"99e7e747ed35f54403f103b8de7844206c0f87552d02a0f4dd999cef7b723cd7"} Nov 03 21:12:39.742951 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:39.742882 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-2f36ee-zt8xb" podStartSLOduration=1.742861105 podStartE2EDuration="1.742861105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:12:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:12:39.738170139 +0000 UTC m=+1994.963489951" watchObservedRunningTime="2026-11-03 21:12:39.742861105 +0000 UTC m=+1994.968180898" Nov 03 21:12:39.838388 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:39.838361 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-70fe43-4d9xj" Nov 03 21:12:39.976949 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:39.976909 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ed41942-1aee-4af5-aff7-96f823fcebff-kube-api-access-9mjxb\" (UniqueName: \"kubernetes.io/projected/4ed41942-1aee-4af5-aff7-96f823fcebff-kube-api-access-9mjxb\") pod \"4ed41942-1aee-4af5-aff7-96f823fcebff\" (UID: \"4ed41942-1aee-4af5-aff7-96f823fcebff\") " Nov 03 21:12:39.977158 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:39.976963 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ed41942-1aee-4af5-aff7-96f823fcebff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ed41942-1aee-4af5-aff7-96f823fcebff-maas-api-openshift-ca\") pod \"4ed41942-1aee-4af5-aff7-96f823fcebff\" (UID: \"4ed41942-1aee-4af5-aff7-96f823fcebff\") " Nov 03 21:12:39.977158 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:39.977082 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ed41942-1aee-4af5-aff7-96f823fcebff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ed41942-1aee-4af5-aff7-96f823fcebff-maas-api-xks-ca\") pod \"4ed41942-1aee-4af5-aff7-96f823fcebff\" (UID: \"4ed41942-1aee-4af5-aff7-96f823fcebff\") " Nov 03 21:12:39.978105 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:39.977654 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ed41942-1aee-4af5-aff7-96f823fcebff-maas-api-openshift-ca" pod "4ed41942-1aee-4af5-aff7-96f823fcebff" (UID: "4ed41942-1aee-4af5-aff7-96f823fcebff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:12:39.978105 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:39.977794 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ed41942-1aee-4af5-aff7-96f823fcebff-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:12:39.980221 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:39.980149 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ed41942-1aee-4af5-aff7-96f823fcebff-kube-api-access-9mjxb" pod "4ed41942-1aee-4af5-aff7-96f823fcebff" (UID: "4ed41942-1aee-4af5-aff7-96f823fcebff"). InnerVolumeSpecName "kube-api-access-9mjxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:12:39.981417 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:39.981389 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ed41942-1aee-4af5-aff7-96f823fcebff-maas-api-xks-ca" pod "4ed41942-1aee-4af5-aff7-96f823fcebff" (UID: "4ed41942-1aee-4af5-aff7-96f823fcebff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:12:40.078861 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:40.078821 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ed41942-1aee-4af5-aff7-96f823fcebff-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:12:40.078861 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:40.078857 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mjxb\" (UniqueName: \"kubernetes.io/projected/4ed41942-1aee-4af5-aff7-96f823fcebff-kube-api-access-9mjxb\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:12:40.630136 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:40.630101 4305 generic.go:356] "Generic (PLEG): container finished" podID="ba7962e8-e17e-4579-bdb8-5b26fa9fb39f" containerID="4e25d72425f9f2dd7535c1b42e421decd8efe591593189de2b7ff301599a94f7" exitCode=0 Nov 03 21:12:40.630592 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:40.630191 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-2f36ee-zt8xb" event={"ID":"ba7962e8-e17e-4579-bdb8-5b26fa9fb39f","Type":"ContainerDied","Data":"4e25d72425f9f2dd7535c1b42e421decd8efe591593189de2b7ff301599a94f7"} Nov 03 21:12:40.631649 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:40.631630 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-70fe43-4d9xj" Nov 03 21:12:40.631795 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:40.631633 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-70fe43-4d9xj" event={"ID":"4ed41942-1aee-4af5-aff7-96f823fcebff","Type":"ContainerDied","Data":"90cbdd8e12d212f313af4b2ff4c92770fad8d041e5bb6ab55243ba89146c1584"} Nov 03 21:12:40.631795 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:40.631740 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90cbdd8e12d212f313af4b2ff4c92770fad8d041e5bb6ab55243ba89146c1584" Nov 03 21:12:41.202794 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:41.202752 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d01cef-ccrrf"] Nov 03 21:12:41.203467 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:41.203446 4305 cpu_manager.go:414] "containerMap: removing container" podUID="4ed41942-1aee-4af5-aff7-96f823fcebff" containerName="revoke-keys" Nov 03 21:12:41.203467 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:41.203463 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ed41942-1aee-4af5-aff7-96f823fcebff" containerName="revoke-keys" Nov 03 21:12:41.203599 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:41.203590 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ed41942-1aee-4af5-aff7-96f823fcebff" containerName="revoke-keys" Nov 03 21:12:41.206699 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:41.206682 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d01cef-ccrrf" Nov 03 21:12:41.266318 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:41.266282 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d01cef-ccrrf"] Nov 03 21:12:41.311598 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:41.311564 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1e68e0d-aaf7-4c44-8419-5e459ebcfc1c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-d01cef-ccrrf\" (UID: \"a1e68e0d-aaf7-4c44-8419-5e459ebcfc1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d01cef-ccrrf" Nov 03 21:12:41.311777 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:41.311609 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1e68e0d-aaf7-4c44-8419-5e459ebcfc1c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-d01cef-ccrrf\" (UID: \"a1e68e0d-aaf7-4c44-8419-5e459ebcfc1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d01cef-ccrrf" Nov 03 21:12:41.311820 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:41.311775 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw4pv\" (UniqueName: \"kubernetes.io/projected/a1e68e0d-aaf7-4c44-8419-5e459ebcfc1c-kube-api-access-zw4pv\") pod \"maas-api-revoke-keys-e2e-worker-w1-d01cef-ccrrf\" (UID: \"a1e68e0d-aaf7-4c44-8419-5e459ebcfc1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d01cef-ccrrf" Nov 03 21:12:41.413950 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:41.413900 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw4pv\" (UniqueName: \"kubernetes.io/projected/a1e68e0d-aaf7-4c44-8419-5e459ebcfc1c-kube-api-access-zw4pv\") pod \"maas-api-revoke-keys-e2e-worker-w1-d01cef-ccrrf\" (UID: \"a1e68e0d-aaf7-4c44-8419-5e459ebcfc1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d01cef-ccrrf" Nov 03 21:12:41.414154 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:41.414130 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1e68e0d-aaf7-4c44-8419-5e459ebcfc1c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-d01cef-ccrrf\" (UID: \"a1e68e0d-aaf7-4c44-8419-5e459ebcfc1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d01cef-ccrrf" Nov 03 21:12:41.414271 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:41.414159 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1e68e0d-aaf7-4c44-8419-5e459ebcfc1c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-d01cef-ccrrf\" (UID: \"a1e68e0d-aaf7-4c44-8419-5e459ebcfc1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d01cef-ccrrf" Nov 03 21:12:41.414806 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:41.414787 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1e68e0d-aaf7-4c44-8419-5e459ebcfc1c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-d01cef-ccrrf\" (UID: \"a1e68e0d-aaf7-4c44-8419-5e459ebcfc1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d01cef-ccrrf" Nov 03 21:12:41.416764 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:41.416741 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1e68e0d-aaf7-4c44-8419-5e459ebcfc1c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-d01cef-ccrrf\" (UID: \"a1e68e0d-aaf7-4c44-8419-5e459ebcfc1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d01cef-ccrrf" Nov 03 21:12:41.476640 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:41.476566 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw4pv\" (UniqueName: \"kubernetes.io/projected/a1e68e0d-aaf7-4c44-8419-5e459ebcfc1c-kube-api-access-zw4pv\") pod \"maas-api-revoke-keys-e2e-worker-w1-d01cef-ccrrf\" (UID: \"a1e68e0d-aaf7-4c44-8419-5e459ebcfc1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d01cef-ccrrf" Nov 03 21:12:41.520599 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:41.520562 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d01cef-ccrrf" Nov 03 21:12:41.793287 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:41.793261 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-2f36ee-zt8xb" Nov 03 21:12:41.905473 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:41.905424 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d01cef-ccrrf"] Nov 03 21:12:41.905983 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:12:41.905962 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1e68e0d_aaf7_4c44_8419_5e459ebcfc1c.slice/crio-cf65d55ab45800f402dab439fe5171815d94c5c890c06c878e143f5f27ce3613 WatchSource:0}: Error finding container cf65d55ab45800f402dab439fe5171815d94c5c890c06c878e143f5f27ce3613: Status 404 returned error can't find the container with id cf65d55ab45800f402dab439fe5171815d94c5c890c06c878e143f5f27ce3613 Nov 03 21:12:41.920344 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:41.920317 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba7962e8-e17e-4579-bdb8-5b26fa9fb39f-kube-api-access-c8bsj\" (UniqueName: \"kubernetes.io/projected/ba7962e8-e17e-4579-bdb8-5b26fa9fb39f-kube-api-access-c8bsj\") pod \"ba7962e8-e17e-4579-bdb8-5b26fa9fb39f\" (UID: \"ba7962e8-e17e-4579-bdb8-5b26fa9fb39f\") " Nov 03 21:12:41.920495 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:41.920400 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba7962e8-e17e-4579-bdb8-5b26fa9fb39f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba7962e8-e17e-4579-bdb8-5b26fa9fb39f-maas-api-openshift-ca\") pod \"ba7962e8-e17e-4579-bdb8-5b26fa9fb39f\" (UID: \"ba7962e8-e17e-4579-bdb8-5b26fa9fb39f\") " Nov 03 21:12:41.920651 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:41.920631 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba7962e8-e17e-4579-bdb8-5b26fa9fb39f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba7962e8-e17e-4579-bdb8-5b26fa9fb39f-maas-api-xks-ca\") pod \"ba7962e8-e17e-4579-bdb8-5b26fa9fb39f\" (UID: \"ba7962e8-e17e-4579-bdb8-5b26fa9fb39f\") " Nov 03 21:12:41.920796 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:41.920767 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba7962e8-e17e-4579-bdb8-5b26fa9fb39f-maas-api-openshift-ca" pod "ba7962e8-e17e-4579-bdb8-5b26fa9fb39f" (UID: "ba7962e8-e17e-4579-bdb8-5b26fa9fb39f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:12:41.921207 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:41.921184 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba7962e8-e17e-4579-bdb8-5b26fa9fb39f-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:12:41.922721 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:41.922674 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba7962e8-e17e-4579-bdb8-5b26fa9fb39f-kube-api-access-c8bsj" pod "ba7962e8-e17e-4579-bdb8-5b26fa9fb39f" (UID: "ba7962e8-e17e-4579-bdb8-5b26fa9fb39f"). InnerVolumeSpecName "kube-api-access-c8bsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:12:41.922781 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:41.922747 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba7962e8-e17e-4579-bdb8-5b26fa9fb39f-maas-api-xks-ca" pod "ba7962e8-e17e-4579-bdb8-5b26fa9fb39f" (UID: "ba7962e8-e17e-4579-bdb8-5b26fa9fb39f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:12:42.023315 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:42.023229 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba7962e8-e17e-4579-bdb8-5b26fa9fb39f-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:12:42.023315 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:42.023271 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8bsj\" (UniqueName: \"kubernetes.io/projected/ba7962e8-e17e-4579-bdb8-5b26fa9fb39f-kube-api-access-c8bsj\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:12:42.647667 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:42.647630 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d01cef-ccrrf" event={"ID":"a1e68e0d-aaf7-4c44-8419-5e459ebcfc1c","Type":"ContainerStarted","Data":"372b7fa03e7ff0ba314086a410a630013329a2bb1896c82b760559d5ef3aaf0a"} Nov 03 21:12:42.647865 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:42.647690 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d01cef-ccrrf" event={"ID":"a1e68e0d-aaf7-4c44-8419-5e459ebcfc1c","Type":"ContainerStarted","Data":"cf65d55ab45800f402dab439fe5171815d94c5c890c06c878e143f5f27ce3613"} Nov 03 21:12:42.649186 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:42.649161 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-2f36ee-zt8xb" Nov 03 21:12:42.649327 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:42.649158 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-2f36ee-zt8xb" event={"ID":"ba7962e8-e17e-4579-bdb8-5b26fa9fb39f","Type":"ContainerDied","Data":"99e7e747ed35f54403f103b8de7844206c0f87552d02a0f4dd999cef7b723cd7"} Nov 03 21:12:42.649327 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:42.649290 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99e7e747ed35f54403f103b8de7844206c0f87552d02a0f4dd999cef7b723cd7" Nov 03 21:12:42.765201 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:42.765096 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d01cef-ccrrf" podStartSLOduration=1.7650815309999999 podStartE2EDuration="1.765081531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:12:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:12:42.762026194 +0000 UTC m=+1997.987345996" watchObservedRunningTime="2026-11-03 21:12:42.765081531 +0000 UTC m=+1997.990401321" Nov 03 21:12:43.657924 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:43.657884 4305 generic.go:356] "Generic (PLEG): container finished" podID="a1e68e0d-aaf7-4c44-8419-5e459ebcfc1c" containerID="372b7fa03e7ff0ba314086a410a630013329a2bb1896c82b760559d5ef3aaf0a" exitCode=0 Nov 03 21:12:43.658341 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:43.657962 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d01cef-ccrrf" event={"ID":"a1e68e0d-aaf7-4c44-8419-5e459ebcfc1c","Type":"ContainerDied","Data":"372b7fa03e7ff0ba314086a410a630013329a2bb1896c82b760559d5ef3aaf0a"} Nov 03 21:12:44.856659 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:44.856630 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d01cef-ccrrf" Nov 03 21:12:44.970075 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:44.969995 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1e68e0d-aaf7-4c44-8419-5e459ebcfc1c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1e68e0d-aaf7-4c44-8419-5e459ebcfc1c-maas-api-openshift-ca\") pod \"a1e68e0d-aaf7-4c44-8419-5e459ebcfc1c\" (UID: \"a1e68e0d-aaf7-4c44-8419-5e459ebcfc1c\") " Nov 03 21:12:44.970215 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:44.970102 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1e68e0d-aaf7-4c44-8419-5e459ebcfc1c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1e68e0d-aaf7-4c44-8419-5e459ebcfc1c-maas-api-xks-ca\") pod \"a1e68e0d-aaf7-4c44-8419-5e459ebcfc1c\" (UID: \"a1e68e0d-aaf7-4c44-8419-5e459ebcfc1c\") " Nov 03 21:12:44.970215 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:44.970167 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1e68e0d-aaf7-4c44-8419-5e459ebcfc1c-kube-api-access-zw4pv\" (UniqueName: \"kubernetes.io/projected/a1e68e0d-aaf7-4c44-8419-5e459ebcfc1c-kube-api-access-zw4pv\") pod \"a1e68e0d-aaf7-4c44-8419-5e459ebcfc1c\" (UID: \"a1e68e0d-aaf7-4c44-8419-5e459ebcfc1c\") " Nov 03 21:12:44.972546 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:44.970811 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1e68e0d-aaf7-4c44-8419-5e459ebcfc1c-maas-api-openshift-ca" pod "a1e68e0d-aaf7-4c44-8419-5e459ebcfc1c" (UID: "a1e68e0d-aaf7-4c44-8419-5e459ebcfc1c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:12:44.972801 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:44.972776 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1e68e0d-aaf7-4c44-8419-5e459ebcfc1c-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:12:44.976484 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:44.973835 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1e68e0d-aaf7-4c44-8419-5e459ebcfc1c-kube-api-access-zw4pv" pod "a1e68e0d-aaf7-4c44-8419-5e459ebcfc1c" (UID: "a1e68e0d-aaf7-4c44-8419-5e459ebcfc1c"). InnerVolumeSpecName "kube-api-access-zw4pv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:12:44.982999 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:44.982942 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1e68e0d-aaf7-4c44-8419-5e459ebcfc1c-maas-api-xks-ca" pod "a1e68e0d-aaf7-4c44-8419-5e459ebcfc1c" (UID: "a1e68e0d-aaf7-4c44-8419-5e459ebcfc1c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:12:45.074113 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:45.074077 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1e68e0d-aaf7-4c44-8419-5e459ebcfc1c-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:12:45.074113 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:45.074110 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zw4pv\" (UniqueName: \"kubernetes.io/projected/a1e68e0d-aaf7-4c44-8419-5e459ebcfc1c-kube-api-access-zw4pv\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:12:45.635410 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:45.635362 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26"] Nov 03 21:12:45.640457 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:45.640404 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt"] Nov 03 21:12:45.674504 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:45.674473 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d01cef-ccrrf" Nov 03 21:12:45.674504 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:45.674466 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d01cef-ccrrf" event={"ID":"a1e68e0d-aaf7-4c44-8419-5e459ebcfc1c","Type":"ContainerDied","Data":"cf65d55ab45800f402dab439fe5171815d94c5c890c06c878e143f5f27ce3613"} Nov 03 21:12:45.674739 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:45.674578 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf65d55ab45800f402dab439fe5171815d94c5c890c06c878e143f5f27ce3613" Nov 03 21:12:46.322722 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:12:46.322680 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:12:53.055420 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:53.055384 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0236317-jcldz"] Nov 03 21:12:53.105136 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:53.105094 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0236317-jcldz"] Nov 03 21:12:54.896491 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:54.896458 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6b1772b-0599-4077-a0df-c21e82e80d97" path="/var/lib/kubelet/pods/b6b1772b-0599-4077-a0df-c21e82e80d97/volumes" Nov 03 21:12:55.057462 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:55.057416 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-d01cef-996dcbc66-xfqll"] Nov 03 21:12:55.171318 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:55.171233 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-d01cef-848dd5d487-swhfc"] Nov 03 21:12:56.715011 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:56.714970 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b8ac68-hqlqx"] Nov 03 21:12:56.715928 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:56.715896 4305 cpu_manager.go:414] "containerMap: removing container" podUID="ba7962e8-e17e-4579-bdb8-5b26fa9fb39f" containerName="revoke-keys" Nov 03 21:12:56.716039 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:56.715930 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba7962e8-e17e-4579-bdb8-5b26fa9fb39f" containerName="revoke-keys" Nov 03 21:12:56.716039 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:56.716016 4305 cpu_manager.go:414] "containerMap: removing container" podUID="a1e68e0d-aaf7-4c44-8419-5e459ebcfc1c" containerName="revoke-keys" Nov 03 21:12:56.716039 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:56.716026 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1e68e0d-aaf7-4c44-8419-5e459ebcfc1c" containerName="revoke-keys" Nov 03 21:12:56.716178 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:56.716164 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1e68e0d-aaf7-4c44-8419-5e459ebcfc1c" containerName="revoke-keys" Nov 03 21:12:56.716224 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:56.716181 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba7962e8-e17e-4579-bdb8-5b26fa9fb39f" containerName="revoke-keys" Nov 03 21:12:56.720089 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:56.720069 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b8ac68-hqlqx" Nov 03 21:12:56.779270 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:56.779237 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b8ac68-hqlqx"] Nov 03 21:12:56.790238 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:56.790193 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt486\" (UniqueName: \"kubernetes.io/projected/d6b6e186-36b1-4e38-9ca2-187dc868fc2b-kube-api-access-wt486\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b8ac68-hqlqx\" (UID: \"d6b6e186-36b1-4e38-9ca2-187dc868fc2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b8ac68-hqlqx" Nov 03 21:12:56.790387 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:56.790362 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6b6e186-36b1-4e38-9ca2-187dc868fc2b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b8ac68-hqlqx\" (UID: \"d6b6e186-36b1-4e38-9ca2-187dc868fc2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b8ac68-hqlqx" Nov 03 21:12:56.790477 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:56.790404 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6b6e186-36b1-4e38-9ca2-187dc868fc2b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b8ac68-hqlqx\" (UID: \"d6b6e186-36b1-4e38-9ca2-187dc868fc2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b8ac68-hqlqx" Nov 03 21:12:56.823856 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:56.823822 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77vtd\"" Nov 03 21:12:56.823856 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:56.823835 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:12:56.824181 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:56.823904 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:12:56.826616 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:56.826596 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:12:56.891346 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:56.891310 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt486\" (UniqueName: \"kubernetes.io/projected/d6b6e186-36b1-4e38-9ca2-187dc868fc2b-kube-api-access-wt486\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b8ac68-hqlqx\" (UID: \"d6b6e186-36b1-4e38-9ca2-187dc868fc2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b8ac68-hqlqx" Nov 03 21:12:56.891578 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:56.891414 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6b6e186-36b1-4e38-9ca2-187dc868fc2b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b8ac68-hqlqx\" (UID: \"d6b6e186-36b1-4e38-9ca2-187dc868fc2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b8ac68-hqlqx" Nov 03 21:12:56.891578 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:56.891497 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6b6e186-36b1-4e38-9ca2-187dc868fc2b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b8ac68-hqlqx\" (UID: \"d6b6e186-36b1-4e38-9ca2-187dc868fc2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b8ac68-hqlqx" Nov 03 21:12:56.893156 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:56.893106 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6b6e186-36b1-4e38-9ca2-187dc868fc2b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b8ac68-hqlqx\" (UID: \"d6b6e186-36b1-4e38-9ca2-187dc868fc2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b8ac68-hqlqx" Nov 03 21:12:56.894889 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:56.894844 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6b6e186-36b1-4e38-9ca2-187dc868fc2b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b8ac68-hqlqx\" (UID: \"d6b6e186-36b1-4e38-9ca2-187dc868fc2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b8ac68-hqlqx" Nov 03 21:12:56.975558 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:56.963412 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt486\" (UniqueName: \"kubernetes.io/projected/d6b6e186-36b1-4e38-9ca2-187dc868fc2b-kube-api-access-wt486\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b8ac68-hqlqx\" (UID: \"d6b6e186-36b1-4e38-9ca2-187dc868fc2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b8ac68-hqlqx" Nov 03 21:12:57.035224 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:57.035180 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b8ac68-hqlqx" Nov 03 21:12:57.427535 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:57.427508 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b8ac68-hqlqx"] Nov 03 21:12:57.428815 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:12:57.428773 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6b6e186_36b1_4e38_9ca2_187dc868fc2b.slice/crio-4e9ba10cac4a8f42a8f2dc6f299a49fabc05609d1e27ce118c06634fc75125fa WatchSource:0}: Error finding container 4e9ba10cac4a8f42a8f2dc6f299a49fabc05609d1e27ce118c06634fc75125fa: Status 404 returned error can't find the container with id 4e9ba10cac4a8f42a8f2dc6f299a49fabc05609d1e27ce118c06634fc75125fa Nov 03 21:12:57.764933 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:57.764901 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b8ac68-hqlqx" event={"ID":"d6b6e186-36b1-4e38-9ca2-187dc868fc2b","Type":"ContainerStarted","Data":"4e9ba10cac4a8f42a8f2dc6f299a49fabc05609d1e27ce118c06634fc75125fa"} Nov 03 21:12:57.987234 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:57.987148 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07550c72-52hqf"] Nov 03 21:12:57.991680 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:57.991660 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07550c72-52hqf" Nov 03 21:12:58.015016 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:58.014984 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrb6n\" (UniqueName: \"kubernetes.io/projected/8a5f2262-0ce7-4ad7-89f0-a1145d1da1dd-kube-api-access-lrb6n\") pod \"maas-api-revoke-sub-07550c72-52hqf\" (UID: \"8a5f2262-0ce7-4ad7-89f0-a1145d1da1dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07550c72-52hqf" Nov 03 21:12:58.015221 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:58.015042 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a5f2262-0ce7-4ad7-89f0-a1145d1da1dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07550c72-52hqf\" (UID: \"8a5f2262-0ce7-4ad7-89f0-a1145d1da1dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07550c72-52hqf" Nov 03 21:12:58.015297 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:58.015242 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a5f2262-0ce7-4ad7-89f0-a1145d1da1dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07550c72-52hqf\" (UID: \"8a5f2262-0ce7-4ad7-89f0-a1145d1da1dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07550c72-52hqf" Nov 03 21:12:58.017172 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:58.017145 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:12:58.048487 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:58.048449 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07550c72-52hqf"] Nov 03 21:12:58.080895 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:58.080863 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0944521f-8t75q"] Nov 03 21:12:58.085273 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:58.085251 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0944521f-8t75q" Nov 03 21:12:58.116546 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:58.116515 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22p8b\" (UniqueName: \"kubernetes.io/projected/5448207d-7597-4ebd-ad24-913b6564f5ad-kube-api-access-22p8b\") pod \"maas-api-revoke-sub-0944521f-8t75q\" (UID: \"5448207d-7597-4ebd-ad24-913b6564f5ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0944521f-8t75q" Nov 03 21:12:58.116734 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:58.116555 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrb6n\" (UniqueName: \"kubernetes.io/projected/8a5f2262-0ce7-4ad7-89f0-a1145d1da1dd-kube-api-access-lrb6n\") pod \"maas-api-revoke-sub-07550c72-52hqf\" (UID: \"8a5f2262-0ce7-4ad7-89f0-a1145d1da1dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07550c72-52hqf" Nov 03 21:12:58.116734 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:58.116598 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a5f2262-0ce7-4ad7-89f0-a1145d1da1dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07550c72-52hqf\" (UID: \"8a5f2262-0ce7-4ad7-89f0-a1145d1da1dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07550c72-52hqf" Nov 03 21:12:58.116734 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:58.116677 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5448207d-7597-4ebd-ad24-913b6564f5ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0944521f-8t75q\" (UID: \"5448207d-7597-4ebd-ad24-913b6564f5ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0944521f-8t75q" Nov 03 21:12:58.116941 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:58.116869 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a5f2262-0ce7-4ad7-89f0-a1145d1da1dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07550c72-52hqf\" (UID: \"8a5f2262-0ce7-4ad7-89f0-a1145d1da1dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07550c72-52hqf" Nov 03 21:12:58.117157 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:58.117014 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5448207d-7597-4ebd-ad24-913b6564f5ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0944521f-8t75q\" (UID: \"5448207d-7597-4ebd-ad24-913b6564f5ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0944521f-8t75q" Nov 03 21:12:58.117664 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:58.117641 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a5f2262-0ce7-4ad7-89f0-a1145d1da1dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07550c72-52hqf\" (UID: \"8a5f2262-0ce7-4ad7-89f0-a1145d1da1dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07550c72-52hqf" Nov 03 21:12:58.119800 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:58.119775 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a5f2262-0ce7-4ad7-89f0-a1145d1da1dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07550c72-52hqf\" (UID: \"8a5f2262-0ce7-4ad7-89f0-a1145d1da1dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07550c72-52hqf" Nov 03 21:12:58.142057 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:58.141964 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0944521f-8t75q"] Nov 03 21:12:58.205229 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:58.205184 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrb6n\" (UniqueName: \"kubernetes.io/projected/8a5f2262-0ce7-4ad7-89f0-a1145d1da1dd-kube-api-access-lrb6n\") pod \"maas-api-revoke-sub-07550c72-52hqf\" (UID: \"8a5f2262-0ce7-4ad7-89f0-a1145d1da1dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07550c72-52hqf" Nov 03 21:12:58.219209 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:58.219176 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5448207d-7597-4ebd-ad24-913b6564f5ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0944521f-8t75q\" (UID: \"5448207d-7597-4ebd-ad24-913b6564f5ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0944521f-8t75q" Nov 03 21:12:58.219361 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:58.219257 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22p8b\" (UniqueName: \"kubernetes.io/projected/5448207d-7597-4ebd-ad24-913b6564f5ad-kube-api-access-22p8b\") pod \"maas-api-revoke-sub-0944521f-8t75q\" (UID: \"5448207d-7597-4ebd-ad24-913b6564f5ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0944521f-8t75q" Nov 03 21:12:58.219361 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:58.219333 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5448207d-7597-4ebd-ad24-913b6564f5ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0944521f-8t75q\" (UID: \"5448207d-7597-4ebd-ad24-913b6564f5ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0944521f-8t75q" Nov 03 21:12:58.220273 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:58.220244 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5448207d-7597-4ebd-ad24-913b6564f5ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0944521f-8t75q\" (UID: \"5448207d-7597-4ebd-ad24-913b6564f5ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0944521f-8t75q" Nov 03 21:12:58.222958 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:58.222934 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5448207d-7597-4ebd-ad24-913b6564f5ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0944521f-8t75q\" (UID: \"5448207d-7597-4ebd-ad24-913b6564f5ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0944521f-8t75q" Nov 03 21:12:58.281532 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:58.281414 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22p8b\" (UniqueName: \"kubernetes.io/projected/5448207d-7597-4ebd-ad24-913b6564f5ad-kube-api-access-22p8b\") pod \"maas-api-revoke-sub-0944521f-8t75q\" (UID: \"5448207d-7597-4ebd-ad24-913b6564f5ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0944521f-8t75q" Nov 03 21:12:58.307229 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:58.307199 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07550c72-52hqf" Nov 03 21:12:58.401743 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:58.401695 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0944521f-8t75q" Nov 03 21:12:58.724312 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:12:58.724277 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a5f2262_0ce7_4ad7_89f0_a1145d1da1dd.slice/crio-e40d9841bc94f08efa6b493e6cd38ebe6cf77dc58977b894f5d5a62232570a62 WatchSource:0}: Error finding container e40d9841bc94f08efa6b493e6cd38ebe6cf77dc58977b894f5d5a62232570a62: Status 404 returned error can't find the container with id e40d9841bc94f08efa6b493e6cd38ebe6cf77dc58977b894f5d5a62232570a62 Nov 03 21:12:58.726081 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:58.726041 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07550c72-52hqf"] Nov 03 21:12:58.774086 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:58.774048 4305 generic.go:356] "Generic (PLEG): container finished" podID="d6b6e186-36b1-4e38-9ca2-187dc868fc2b" containerID="0ec7145920fb66e107dd59a3dcee6480190d33cf572d9619856ed24c712964bd" exitCode=0 Nov 03 21:12:58.774587 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:58.774130 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b8ac68-hqlqx" event={"ID":"d6b6e186-36b1-4e38-9ca2-187dc868fc2b","Type":"ContainerDied","Data":"0ec7145920fb66e107dd59a3dcee6480190d33cf572d9619856ed24c712964bd"} Nov 03 21:12:58.775803 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:58.775780 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07550c72-52hqf" event={"ID":"8a5f2262-0ce7-4ad7-89f0-a1145d1da1dd","Type":"ContainerStarted","Data":"e40d9841bc94f08efa6b493e6cd38ebe6cf77dc58977b894f5d5a62232570a62"} Nov 03 21:12:58.839486 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:58.839456 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0944521f-8t75q"] Nov 03 21:12:58.840118 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:12:58.840091 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5448207d_7597_4ebd_ad24_913b6564f5ad.slice/crio-d4cdfb821d25bbaeed7b327974111043d45ea9e40fb56f3ef1bd73ae87bab763 WatchSource:0}: Error finding container d4cdfb821d25bbaeed7b327974111043d45ea9e40fb56f3ef1bd73ae87bab763: Status 404 returned error can't find the container with id d4cdfb821d25bbaeed7b327974111043d45ea9e40fb56f3ef1bd73ae87bab763 Nov 03 21:12:58.869374 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:58.869344 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-2f36ee-zt8xb"] Nov 03 21:12:58.887305 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:58.887225 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-2f36ee-zt8xb"] Nov 03 21:12:58.896512 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:58.896480 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba7962e8-e17e-4579-bdb8-5b26fa9fb39f" path="/var/lib/kubelet/pods/ba7962e8-e17e-4579-bdb8-5b26fa9fb39f/volumes" Nov 03 21:12:59.785057 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:59.785024 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0944521f-8t75q" event={"ID":"5448207d-7597-4ebd-ad24-913b6564f5ad","Type":"ContainerStarted","Data":"b631d629c3e97e14ca9b4ac44b9f2e22484db843dda7e6bb5831f19f07855524"} Nov 03 21:12:59.785540 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:59.785087 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0944521f-8t75q" event={"ID":"5448207d-7597-4ebd-ad24-913b6564f5ad","Type":"ContainerStarted","Data":"d4cdfb821d25bbaeed7b327974111043d45ea9e40fb56f3ef1bd73ae87bab763"} Nov 03 21:12:59.786773 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:59.786743 4305 generic.go:356] "Generic (PLEG): container finished" podID="8a5f2262-0ce7-4ad7-89f0-a1145d1da1dd" containerID="8c8bcd4e3c09e5f9a6d17936d53669c10a0038153ee3029a1a6ef9259f6de8c9" exitCode=0 Nov 03 21:12:59.786914 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:59.786881 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07550c72-52hqf" event={"ID":"8a5f2262-0ce7-4ad7-89f0-a1145d1da1dd","Type":"ContainerDied","Data":"8c8bcd4e3c09e5f9a6d17936d53669c10a0038153ee3029a1a6ef9259f6de8c9"} Nov 03 21:12:59.910457 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:59.909786 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0944521f-8t75q" podStartSLOduration=1.909764947 podStartE2EDuration="1.909764947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:12:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:12:59.900901299 +0000 UTC m=+2015.126221110" watchObservedRunningTime="2026-11-03 21:12:59.909764947 +0000 UTC m=+2015.135084742" Nov 03 21:12:59.963327 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:59.963259 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b8ac68-hqlqx" Nov 03 21:12:59.983795 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:59.983763 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6b6e186-36b1-4e38-9ca2-187dc868fc2b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6b6e186-36b1-4e38-9ca2-187dc868fc2b-maas-api-openshift-ca\") pod \"d6b6e186-36b1-4e38-9ca2-187dc868fc2b\" (UID: \"d6b6e186-36b1-4e38-9ca2-187dc868fc2b\") " Nov 03 21:12:59.983981 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:59.983806 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6b6e186-36b1-4e38-9ca2-187dc868fc2b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6b6e186-36b1-4e38-9ca2-187dc868fc2b-maas-api-xks-ca\") pod \"d6b6e186-36b1-4e38-9ca2-187dc868fc2b\" (UID: \"d6b6e186-36b1-4e38-9ca2-187dc868fc2b\") " Nov 03 21:12:59.983981 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:59.983870 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6b6e186-36b1-4e38-9ca2-187dc868fc2b-kube-api-access-wt486\" (UniqueName: \"kubernetes.io/projected/d6b6e186-36b1-4e38-9ca2-187dc868fc2b-kube-api-access-wt486\") pod \"d6b6e186-36b1-4e38-9ca2-187dc868fc2b\" (UID: \"d6b6e186-36b1-4e38-9ca2-187dc868fc2b\") " Nov 03 21:12:59.984260 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:59.984230 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6b6e186-36b1-4e38-9ca2-187dc868fc2b-maas-api-openshift-ca" pod "d6b6e186-36b1-4e38-9ca2-187dc868fc2b" (UID: "d6b6e186-36b1-4e38-9ca2-187dc868fc2b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:12:59.984721 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:59.984697 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6b6e186-36b1-4e38-9ca2-187dc868fc2b-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:12:59.986222 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:59.986194 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6b6e186-36b1-4e38-9ca2-187dc868fc2b-maas-api-xks-ca" pod "d6b6e186-36b1-4e38-9ca2-187dc868fc2b" (UID: "d6b6e186-36b1-4e38-9ca2-187dc868fc2b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:12:59.986635 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:12:59.986619 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6b6e186-36b1-4e38-9ca2-187dc868fc2b-kube-api-access-wt486" pod "d6b6e186-36b1-4e38-9ca2-187dc868fc2b" (UID: "d6b6e186-36b1-4e38-9ca2-187dc868fc2b"). InnerVolumeSpecName "kube-api-access-wt486". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:13:00.057765 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:00.057721 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-d01cef-996dcbc66-xfqll" podUID="ffdb1619-de55-4707-8dbf-b40be40c608e" containerName="payload-processing" containerID="cri-o://914a4233ec2fe008a13bfdb1e3698a77b22c3f8e2869726776136fe677111ed5" gracePeriod=25 Nov 03 21:13:00.086256 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:00.086224 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6b6e186-36b1-4e38-9ca2-187dc868fc2b-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:00.086256 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:00.086255 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wt486\" (UniqueName: \"kubernetes.io/projected/d6b6e186-36b1-4e38-9ca2-187dc868fc2b-kube-api-access-wt486\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:00.176501 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:00.172375 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d01cef-848dd5d487-swhfc" podUID="aab129fc-dcb8-4cd0-beaa-49f36dfb3f38" containerName="payload-pre-processing" containerID="cri-o://ae84a85c3c2b8975a14902e1a23cbc07af7b701ed99cf7971ee6e5a902538628" gracePeriod=25 Nov 03 21:13:00.347273 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:00.347248 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-d01cef-996dcbc66-xfqll" Nov 03 21:13:00.390158 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:00.390117 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffdb1619-de55-4707-8dbf-b40be40c608e-kube-api-access-gql4p\" (UniqueName: \"kubernetes.io/projected/ffdb1619-de55-4707-8dbf-b40be40c608e-kube-api-access-gql4p\") pod \"ffdb1619-de55-4707-8dbf-b40be40c608e\" (UID: \"ffdb1619-de55-4707-8dbf-b40be40c608e\") " Nov 03 21:13:00.390509 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:00.390307 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffdb1619-de55-4707-8dbf-b40be40c608e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffdb1619-de55-4707-8dbf-b40be40c608e-plugins-config-volume\") pod \"ffdb1619-de55-4707-8dbf-b40be40c608e\" (UID: \"ffdb1619-de55-4707-8dbf-b40be40c608e\") " Nov 03 21:13:00.393486 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:00.391366 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffdb1619-de55-4707-8dbf-b40be40c608e-plugins-config-volume" pod "ffdb1619-de55-4707-8dbf-b40be40c608e" (UID: "ffdb1619-de55-4707-8dbf-b40be40c608e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:13:00.404278 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:00.404230 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffdb1619-de55-4707-8dbf-b40be40c608e-kube-api-access-gql4p" pod "ffdb1619-de55-4707-8dbf-b40be40c608e" (UID: "ffdb1619-de55-4707-8dbf-b40be40c608e"). InnerVolumeSpecName "kube-api-access-gql4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:13:00.438589 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:00.437732 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-70fe43-4d9xj"] Nov 03 21:13:00.443898 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:00.443865 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-70fe43-4d9xj"] Nov 03 21:13:00.465680 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:00.465649 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d01cef-848dd5d487-swhfc" Nov 03 21:13:00.492141 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:00.492108 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aab129fc-dcb8-4cd0-beaa-49f36dfb3f38-kube-api-access-mgmrn\" (UniqueName: \"kubernetes.io/projected/aab129fc-dcb8-4cd0-beaa-49f36dfb3f38-kube-api-access-mgmrn\") pod \"aab129fc-dcb8-4cd0-beaa-49f36dfb3f38\" (UID: \"aab129fc-dcb8-4cd0-beaa-49f36dfb3f38\") " Nov 03 21:13:00.492406 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:00.492383 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aab129fc-dcb8-4cd0-beaa-49f36dfb3f38-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aab129fc-dcb8-4cd0-beaa-49f36dfb3f38-plugins-config-volume\") pod \"aab129fc-dcb8-4cd0-beaa-49f36dfb3f38\" (UID: \"aab129fc-dcb8-4cd0-beaa-49f36dfb3f38\") " Nov 03 21:13:00.492855 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:00.492824 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aab129fc-dcb8-4cd0-beaa-49f36dfb3f38-plugins-config-volume" pod "aab129fc-dcb8-4cd0-beaa-49f36dfb3f38" (UID: "aab129fc-dcb8-4cd0-beaa-49f36dfb3f38"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:13:00.493087 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:00.493063 4305 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aab129fc-dcb8-4cd0-beaa-49f36dfb3f38-plugins-config-volume\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:00.493164 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:00.493095 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gql4p\" (UniqueName: \"kubernetes.io/projected/ffdb1619-de55-4707-8dbf-b40be40c608e-kube-api-access-gql4p\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:00.493164 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:00.493111 4305 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffdb1619-de55-4707-8dbf-b40be40c608e-plugins-config-volume\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:00.494525 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:00.494495 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aab129fc-dcb8-4cd0-beaa-49f36dfb3f38-kube-api-access-mgmrn" pod "aab129fc-dcb8-4cd0-beaa-49f36dfb3f38" (UID: "aab129fc-dcb8-4cd0-beaa-49f36dfb3f38"). InnerVolumeSpecName "kube-api-access-mgmrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:13:00.595314 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:00.595233 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgmrn\" (UniqueName: \"kubernetes.io/projected/aab129fc-dcb8-4cd0-beaa-49f36dfb3f38-kube-api-access-mgmrn\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:00.659541 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:00.659500 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" podUID="d5d3dcaa-a51a-40c6-8049-871b300552e1" containerName="main" containerID="cri-o://f267d2c6e0195a332ca66497d52a5a5879c135891e181fe0a9c588980f421940" gracePeriod=45 Nov 03 21:13:00.663699 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:00.663659 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" podUID="6e7f30a9-0862-48a5-9ab7-e692e75bb294" containerName="main" containerID="cri-o://0628e6efd4338d498f7287cd26233a950f96353d5c6ec3bc3ce3c285a3039816" gracePeriod=45 Nov 03 21:13:00.745995 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:00.745951 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" podUID="d5d3dcaa-a51a-40c6-8049-871b300552e1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": dial tcp 10.134.0.96:8000: connect: connection refused" Nov 03 21:13:00.796871 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:00.796837 4305 generic.go:356] "Generic (PLEG): container finished" podID="5448207d-7597-4ebd-ad24-913b6564f5ad" containerID="b631d629c3e97e14ca9b4ac44b9f2e22484db843dda7e6bb5831f19f07855524" exitCode=0 Nov 03 21:13:00.797319 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:00.796926 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0944521f-8t75q" event={"ID":"5448207d-7597-4ebd-ad24-913b6564f5ad","Type":"ContainerDied","Data":"b631d629c3e97e14ca9b4ac44b9f2e22484db843dda7e6bb5831f19f07855524"} Nov 03 21:13:00.798912 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:00.798875 4305 generic.go:356] "Generic (PLEG): container finished" podID="6e7f30a9-0862-48a5-9ab7-e692e75bb294" containerID="0628e6efd4338d498f7287cd26233a950f96353d5c6ec3bc3ce3c285a3039816" exitCode=0 Nov 03 21:13:00.799053 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:00.798946 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" event={"ID":"6e7f30a9-0862-48a5-9ab7-e692e75bb294","Type":"ContainerDied","Data":"0628e6efd4338d498f7287cd26233a950f96353d5c6ec3bc3ce3c285a3039816"} Nov 03 21:13:00.800859 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:00.800841 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b8ac68-hqlqx" Nov 03 21:13:00.800999 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:00.800853 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b8ac68-hqlqx" event={"ID":"d6b6e186-36b1-4e38-9ca2-187dc868fc2b","Type":"ContainerDied","Data":"4e9ba10cac4a8f42a8f2dc6f299a49fabc05609d1e27ce118c06634fc75125fa"} Nov 03 21:13:00.800999 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:00.800894 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e9ba10cac4a8f42a8f2dc6f299a49fabc05609d1e27ce118c06634fc75125fa" Nov 03 21:13:00.802808 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:00.802784 4305 generic.go:356] "Generic (PLEG): container finished" podID="aab129fc-dcb8-4cd0-beaa-49f36dfb3f38" containerID="ae84a85c3c2b8975a14902e1a23cbc07af7b701ed99cf7971ee6e5a902538628" exitCode=0 Nov 03 21:13:00.802913 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:00.802844 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d01cef-848dd5d487-swhfc" Nov 03 21:13:00.802913 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:00.802870 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d01cef-848dd5d487-swhfc" event={"ID":"aab129fc-dcb8-4cd0-beaa-49f36dfb3f38","Type":"ContainerDied","Data":"ae84a85c3c2b8975a14902e1a23cbc07af7b701ed99cf7971ee6e5a902538628"} Nov 03 21:13:00.803059 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:00.802919 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-d01cef-848dd5d487-swhfc" event={"ID":"aab129fc-dcb8-4cd0-beaa-49f36dfb3f38","Type":"ContainerDied","Data":"d3ca8133974550493164dd0566b879f55d730d4a44aefa67012f9b68d7d0a632"} Nov 03 21:13:00.803059 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:00.802950 4305 scope.go:122] "RemoveContainer" containerID="ae84a85c3c2b8975a14902e1a23cbc07af7b701ed99cf7971ee6e5a902538628" Nov 03 21:13:00.807237 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:00.807203 4305 generic.go:356] "Generic (PLEG): container finished" podID="ffdb1619-de55-4707-8dbf-b40be40c608e" containerID="914a4233ec2fe008a13bfdb1e3698a77b22c3f8e2869726776136fe677111ed5" exitCode=0 Nov 03 21:13:00.807375 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:00.807310 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-d01cef-996dcbc66-xfqll" event={"ID":"ffdb1619-de55-4707-8dbf-b40be40c608e","Type":"ContainerDied","Data":"914a4233ec2fe008a13bfdb1e3698a77b22c3f8e2869726776136fe677111ed5"} Nov 03 21:13:00.807375 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:00.807363 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-d01cef-996dcbc66-xfqll" event={"ID":"ffdb1619-de55-4707-8dbf-b40be40c608e","Type":"ContainerDied","Data":"484fbbf7bbc427b9847e6defcc5e21094822e867bcdb116d53130933058f530f"} Nov 03 21:13:00.807536 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:00.807457 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-d01cef-996dcbc66-xfqll" Nov 03 21:13:00.812678 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:00.812647 4305 generic.go:356] "Generic (PLEG): container finished" podID="d5d3dcaa-a51a-40c6-8049-871b300552e1" containerID="f267d2c6e0195a332ca66497d52a5a5879c135891e181fe0a9c588980f421940" exitCode=0 Nov 03 21:13:00.812980 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:00.812958 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" event={"ID":"d5d3dcaa-a51a-40c6-8049-871b300552e1","Type":"ContainerDied","Data":"f267d2c6e0195a332ca66497d52a5a5879c135891e181fe0a9c588980f421940"} Nov 03 21:13:00.822560 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:00.820329 4305 scope.go:122] "RemoveContainer" containerID="ae84a85c3c2b8975a14902e1a23cbc07af7b701ed99cf7971ee6e5a902538628" Nov 03 21:13:00.822560 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:13:00.820696 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae84a85c3c2b8975a14902e1a23cbc07af7b701ed99cf7971ee6e5a902538628\": container with ID starting with ae84a85c3c2b8975a14902e1a23cbc07af7b701ed99cf7971ee6e5a902538628 not found: ID does not exist" containerID="ae84a85c3c2b8975a14902e1a23cbc07af7b701ed99cf7971ee6e5a902538628" Nov 03 21:13:00.822560 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:00.820738 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae84a85c3c2b8975a14902e1a23cbc07af7b701ed99cf7971ee6e5a902538628"} err="failed to get container status \"ae84a85c3c2b8975a14902e1a23cbc07af7b701ed99cf7971ee6e5a902538628\": rpc error: code = NotFound desc = could not find container \"ae84a85c3c2b8975a14902e1a23cbc07af7b701ed99cf7971ee6e5a902538628\": container with ID starting with ae84a85c3c2b8975a14902e1a23cbc07af7b701ed99cf7971ee6e5a902538628 not found: ID does not exist" Nov 03 21:13:00.822560 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:00.820767 4305 scope.go:122] "RemoveContainer" containerID="914a4233ec2fe008a13bfdb1e3698a77b22c3f8e2869726776136fe677111ed5" Nov 03 21:13:00.836138 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:00.836103 4305 scope.go:122] "RemoveContainer" containerID="914a4233ec2fe008a13bfdb1e3698a77b22c3f8e2869726776136fe677111ed5" Nov 03 21:13:00.837792 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:13:00.837596 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"914a4233ec2fe008a13bfdb1e3698a77b22c3f8e2869726776136fe677111ed5\": container with ID starting with 914a4233ec2fe008a13bfdb1e3698a77b22c3f8e2869726776136fe677111ed5 not found: ID does not exist" containerID="914a4233ec2fe008a13bfdb1e3698a77b22c3f8e2869726776136fe677111ed5" Nov 03 21:13:00.837792 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:00.837632 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"914a4233ec2fe008a13bfdb1e3698a77b22c3f8e2869726776136fe677111ed5"} err="failed to get container status \"914a4233ec2fe008a13bfdb1e3698a77b22c3f8e2869726776136fe677111ed5\": rpc error: code = NotFound desc = could not find container \"914a4233ec2fe008a13bfdb1e3698a77b22c3f8e2869726776136fe677111ed5\": container with ID starting with 914a4233ec2fe008a13bfdb1e3698a77b22c3f8e2869726776136fe677111ed5 not found: ID does not exist" Nov 03 21:13:00.905484 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:00.905416 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ed41942-1aee-4af5-aff7-96f823fcebff" path="/var/lib/kubelet/pods/4ed41942-1aee-4af5-aff7-96f823fcebff/volumes" Nov 03 21:13:01.009684 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:01.009644 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" Nov 03 21:13:01.013042 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:01.012968 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" Nov 03 21:13:01.039688 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:01.039655 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-d01cef-996dcbc66-xfqll"] Nov 03 21:13:01.043485 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:01.043356 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-d01cef-996dcbc66-xfqll"] Nov 03 21:13:01.072904 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:01.072874 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07550c72-52hqf" Nov 03 21:13:01.104612 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:01.104529 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a5f2262-0ce7-4ad7-89f0-a1145d1da1dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a5f2262-0ce7-4ad7-89f0-a1145d1da1dd-maas-api-openshift-ca\") pod \"8a5f2262-0ce7-4ad7-89f0-a1145d1da1dd\" (UID: \"8a5f2262-0ce7-4ad7-89f0-a1145d1da1dd\") " Nov 03 21:13:01.104786 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:01.104758 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5d3dcaa-a51a-40c6-8049-871b300552e1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5d3dcaa-a51a-40c6-8049-871b300552e1-model-cache\") pod \"d5d3dcaa-a51a-40c6-8049-871b300552e1\" (UID: \"d5d3dcaa-a51a-40c6-8049-871b300552e1\") " Nov 03 21:13:01.105154 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:01.104987 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a5f2262-0ce7-4ad7-89f0-a1145d1da1dd-maas-api-openshift-ca" pod "8a5f2262-0ce7-4ad7-89f0-a1145d1da1dd" (UID: "8a5f2262-0ce7-4ad7-89f0-a1145d1da1dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:13:01.105560 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:01.105093 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5d3dcaa-a51a-40c6-8049-871b300552e1-model-cache" pod "d5d3dcaa-a51a-40c6-8049-871b300552e1" (UID: "d5d3dcaa-a51a-40c6-8049-871b300552e1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:13:01.105651 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:01.105629 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5d3dcaa-a51a-40c6-8049-871b300552e1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5d3dcaa-a51a-40c6-8049-871b300552e1-dshm\") pod \"d5d3dcaa-a51a-40c6-8049-871b300552e1\" (UID: \"d5d3dcaa-a51a-40c6-8049-871b300552e1\") " Nov 03 21:13:01.105705 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:01.105667 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e7f30a9-0862-48a5-9ab7-e692e75bb294-tls-certs\" (UniqueName: \"kubernetes.io/secret/6e7f30a9-0862-48a5-9ab7-e692e75bb294-tls-certs\") pod \"6e7f30a9-0862-48a5-9ab7-e692e75bb294\" (UID: \"6e7f30a9-0862-48a5-9ab7-e692e75bb294\") " Nov 03 21:13:01.105740 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:01.105729 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5d3dcaa-a51a-40c6-8049-871b300552e1-home\" (UniqueName: \"kubernetes.io/empty-dir/d5d3dcaa-a51a-40c6-8049-871b300552e1-home\") pod \"d5d3dcaa-a51a-40c6-8049-871b300552e1\" (UID: \"d5d3dcaa-a51a-40c6-8049-871b300552e1\") " Nov 03 21:13:01.105816 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:01.105802 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e7f30a9-0862-48a5-9ab7-e692e75bb294-home\" (UniqueName: \"kubernetes.io/empty-dir/6e7f30a9-0862-48a5-9ab7-e692e75bb294-home\") pod \"6e7f30a9-0862-48a5-9ab7-e692e75bb294\" (UID: \"6e7f30a9-0862-48a5-9ab7-e692e75bb294\") " Nov 03 21:13:01.105855 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:01.105837 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a5f2262-0ce7-4ad7-89f0-a1145d1da1dd-kube-api-access-lrb6n\" (UniqueName: \"kubernetes.io/projected/8a5f2262-0ce7-4ad7-89f0-a1145d1da1dd-kube-api-access-lrb6n\") pod \"8a5f2262-0ce7-4ad7-89f0-a1145d1da1dd\" (UID: \"8a5f2262-0ce7-4ad7-89f0-a1145d1da1dd\") " Nov 03 21:13:01.105897 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:01.105883 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5d3dcaa-a51a-40c6-8049-871b300552e1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5d3dcaa-a51a-40c6-8049-871b300552e1-tmp-dir\") pod \"d5d3dcaa-a51a-40c6-8049-871b300552e1\" (UID: \"d5d3dcaa-a51a-40c6-8049-871b300552e1\") " Nov 03 21:13:01.105934 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:01.105916 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e7f30a9-0862-48a5-9ab7-e692e75bb294-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e7f30a9-0862-48a5-9ab7-e692e75bb294-model-cache\") pod \"6e7f30a9-0862-48a5-9ab7-e692e75bb294\" (UID: \"6e7f30a9-0862-48a5-9ab7-e692e75bb294\") " Nov 03 21:13:01.105984 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:01.105971 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5d3dcaa-a51a-40c6-8049-871b300552e1-tls-certs\" (UniqueName: \"kubernetes.io/secret/d5d3dcaa-a51a-40c6-8049-871b300552e1-tls-certs\") pod \"d5d3dcaa-a51a-40c6-8049-871b300552e1\" (UID: \"d5d3dcaa-a51a-40c6-8049-871b300552e1\") " Nov 03 21:13:01.106023 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:01.106007 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e7f30a9-0862-48a5-9ab7-e692e75bb294-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e7f30a9-0862-48a5-9ab7-e692e75bb294-tmp-dir\") pod \"6e7f30a9-0862-48a5-9ab7-e692e75bb294\" (UID: \"6e7f30a9-0862-48a5-9ab7-e692e75bb294\") " Nov 03 21:13:01.106090 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:01.106076 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5d3dcaa-a51a-40c6-8049-871b300552e1-kube-api-access-bwgpv\" (UniqueName: \"kubernetes.io/projected/d5d3dcaa-a51a-40c6-8049-871b300552e1-kube-api-access-bwgpv\") pod \"d5d3dcaa-a51a-40c6-8049-871b300552e1\" (UID: \"d5d3dcaa-a51a-40c6-8049-871b300552e1\") " Nov 03 21:13:01.106125 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:01.106115 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e7f30a9-0862-48a5-9ab7-e692e75bb294-kube-api-access-scs4x\" (UniqueName: \"kubernetes.io/projected/6e7f30a9-0862-48a5-9ab7-e692e75bb294-kube-api-access-scs4x\") pod \"6e7f30a9-0862-48a5-9ab7-e692e75bb294\" (UID: \"6e7f30a9-0862-48a5-9ab7-e692e75bb294\") " Nov 03 21:13:01.106178 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:01.106165 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a5f2262-0ce7-4ad7-89f0-a1145d1da1dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a5f2262-0ce7-4ad7-89f0-a1145d1da1dd-maas-api-xks-ca\") pod \"8a5f2262-0ce7-4ad7-89f0-a1145d1da1dd\" (UID: \"8a5f2262-0ce7-4ad7-89f0-a1145d1da1dd\") " Nov 03 21:13:01.106248 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:01.106235 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e7f30a9-0862-48a5-9ab7-e692e75bb294-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e7f30a9-0862-48a5-9ab7-e692e75bb294-dshm\") pod \"6e7f30a9-0862-48a5-9ab7-e692e75bb294\" (UID: \"6e7f30a9-0862-48a5-9ab7-e692e75bb294\") " Nov 03 21:13:01.106978 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:01.106954 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a5f2262-0ce7-4ad7-89f0-a1145d1da1dd-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:01.107036 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:01.106983 4305 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5d3dcaa-a51a-40c6-8049-871b300552e1-model-cache\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:01.107478 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:01.107456 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5d3dcaa-a51a-40c6-8049-871b300552e1-tmp-dir" pod "d5d3dcaa-a51a-40c6-8049-871b300552e1" (UID: "d5d3dcaa-a51a-40c6-8049-871b300552e1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:13:01.107791 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:01.107594 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5d3dcaa-a51a-40c6-8049-871b300552e1-home" pod "d5d3dcaa-a51a-40c6-8049-871b300552e1" (UID: "d5d3dcaa-a51a-40c6-8049-871b300552e1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:13:01.107888 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:01.107604 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e7f30a9-0862-48a5-9ab7-e692e75bb294-tmp-dir" pod "6e7f30a9-0862-48a5-9ab7-e692e75bb294" (UID: "6e7f30a9-0862-48a5-9ab7-e692e75bb294"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:13:01.107991 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:01.107794 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e7f30a9-0862-48a5-9ab7-e692e75bb294-model-cache" pod "6e7f30a9-0862-48a5-9ab7-e692e75bb294" (UID: "6e7f30a9-0862-48a5-9ab7-e692e75bb294"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:13:01.108548 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:01.108382 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e7f30a9-0862-48a5-9ab7-e692e75bb294-home" pod "6e7f30a9-0862-48a5-9ab7-e692e75bb294" (UID: "6e7f30a9-0862-48a5-9ab7-e692e75bb294"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:13:01.111349 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:01.111254 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e7f30a9-0862-48a5-9ab7-e692e75bb294-dshm" pod "6e7f30a9-0862-48a5-9ab7-e692e75bb294" (UID: "6e7f30a9-0862-48a5-9ab7-e692e75bb294"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:13:01.111349 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:01.111268 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a5f2262-0ce7-4ad7-89f0-a1145d1da1dd-kube-api-access-lrb6n" pod "8a5f2262-0ce7-4ad7-89f0-a1145d1da1dd" (UID: "8a5f2262-0ce7-4ad7-89f0-a1145d1da1dd"). InnerVolumeSpecName "kube-api-access-lrb6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:13:01.111349 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:01.111286 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5d3dcaa-a51a-40c6-8049-871b300552e1-dshm" pod "d5d3dcaa-a51a-40c6-8049-871b300552e1" (UID: "d5d3dcaa-a51a-40c6-8049-871b300552e1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:13:01.111349 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:01.111323 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5d3dcaa-a51a-40c6-8049-871b300552e1-tls-certs" pod "d5d3dcaa-a51a-40c6-8049-871b300552e1" (UID: "d5d3dcaa-a51a-40c6-8049-871b300552e1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:13:01.111873 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:01.111853 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e7f30a9-0862-48a5-9ab7-e692e75bb294-kube-api-access-scs4x" pod "6e7f30a9-0862-48a5-9ab7-e692e75bb294" (UID: "6e7f30a9-0862-48a5-9ab7-e692e75bb294"). InnerVolumeSpecName "kube-api-access-scs4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:13:01.112555 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:01.112535 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a5f2262-0ce7-4ad7-89f0-a1145d1da1dd-maas-api-xks-ca" pod "8a5f2262-0ce7-4ad7-89f0-a1145d1da1dd" (UID: "8a5f2262-0ce7-4ad7-89f0-a1145d1da1dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:13:01.113107 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:01.113089 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5d3dcaa-a51a-40c6-8049-871b300552e1-kube-api-access-bwgpv" pod "d5d3dcaa-a51a-40c6-8049-871b300552e1" (UID: "d5d3dcaa-a51a-40c6-8049-871b300552e1"). InnerVolumeSpecName "kube-api-access-bwgpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:13:01.113832 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:01.113804 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e7f30a9-0862-48a5-9ab7-e692e75bb294-tls-certs" pod "6e7f30a9-0862-48a5-9ab7-e692e75bb294" (UID: "6e7f30a9-0862-48a5-9ab7-e692e75bb294"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:13:01.162579 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:01.162543 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-d01cef-848dd5d487-swhfc"] Nov 03 21:13:01.171592 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:01.171561 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-d01cef-848dd5d487-swhfc"] Nov 03 21:13:01.208542 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:01.208500 4305 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5d3dcaa-a51a-40c6-8049-871b300552e1-home\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:01.208542 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:01.208538 4305 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e7f30a9-0862-48a5-9ab7-e692e75bb294-home\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:01.208542 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:01.208550 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrb6n\" (UniqueName: \"kubernetes.io/projected/8a5f2262-0ce7-4ad7-89f0-a1145d1da1dd-kube-api-access-lrb6n\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:01.208816 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:01.208581 4305 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5d3dcaa-a51a-40c6-8049-871b300552e1-tmp-dir\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:01.208816 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:01.208593 4305 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e7f30a9-0862-48a5-9ab7-e692e75bb294-model-cache\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:01.208816 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:01.208602 4305 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5d3dcaa-a51a-40c6-8049-871b300552e1-tls-certs\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:01.208816 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:01.208610 4305 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e7f30a9-0862-48a5-9ab7-e692e75bb294-tmp-dir\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:01.208816 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:01.208620 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwgpv\" (UniqueName: \"kubernetes.io/projected/d5d3dcaa-a51a-40c6-8049-871b300552e1-kube-api-access-bwgpv\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:01.208816 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:01.208629 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scs4x\" (UniqueName: \"kubernetes.io/projected/6e7f30a9-0862-48a5-9ab7-e692e75bb294-kube-api-access-scs4x\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:01.208816 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:01.208638 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a5f2262-0ce7-4ad7-89f0-a1145d1da1dd-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:01.208816 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:01.208646 4305 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e7f30a9-0862-48a5-9ab7-e692e75bb294-dshm\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:01.208816 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:01.208654 4305 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5d3dcaa-a51a-40c6-8049-871b300552e1-dshm\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:01.208816 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:01.208662 4305 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e7f30a9-0862-48a5-9ab7-e692e75bb294-tls-certs\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:01.824299 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:01.824272 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" Nov 03 21:13:01.824858 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:01.824306 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26" event={"ID":"d5d3dcaa-a51a-40c6-8049-871b300552e1","Type":"ContainerDied","Data":"21606bf960577ecfcf3e5eecdc322ed2e1f89849c44a2964dac1931fa4fdb52f"} Nov 03 21:13:01.824858 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:01.824360 4305 scope.go:122] "RemoveContainer" containerID="f267d2c6e0195a332ca66497d52a5a5879c135891e181fe0a9c588980f421940" Nov 03 21:13:01.826156 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:01.825846 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07550c72-52hqf" Nov 03 21:13:01.826156 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:01.825868 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07550c72-52hqf" event={"ID":"8a5f2262-0ce7-4ad7-89f0-a1145d1da1dd","Type":"ContainerDied","Data":"e40d9841bc94f08efa6b493e6cd38ebe6cf77dc58977b894f5d5a62232570a62"} Nov 03 21:13:01.826156 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:01.825915 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e40d9841bc94f08efa6b493e6cd38ebe6cf77dc58977b894f5d5a62232570a62" Nov 03 21:13:01.836492 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:01.836463 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" Nov 03 21:13:01.840185 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:01.836491 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt" event={"ID":"6e7f30a9-0862-48a5-9ab7-e692e75bb294","Type":"ContainerDied","Data":"266df45a42582bd33c725e3cdfb2dc05a86f9b4bd8c1f2df563429af679ee2a5"} Nov 03 21:13:01.844909 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:01.844887 4305 scope.go:122] "RemoveContainer" containerID="0628e6efd4338d498f7287cd26233a950f96353d5c6ec3bc3ce3c285a3039816" Nov 03 21:13:01.992527 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:01.992495 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt"] Nov 03 21:13:01.997074 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:01.997050 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-d01cef/distinct-w1-d01cef-kserve-844b8cd9c9-fw7jt"] Nov 03 21:13:02.001420 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:02.001396 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0944521f-8t75q" Nov 03 21:13:02.101813 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:02.101711 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26"] Nov 03 21:13:02.105365 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:02.105335 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-d01cef/distinct-2-w1-d01cef-kserve-547bffcc96-scw26"] Nov 03 21:13:02.120343 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:02.120308 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5448207d-7597-4ebd-ad24-913b6564f5ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5448207d-7597-4ebd-ad24-913b6564f5ad-maas-api-xks-ca\") pod \"5448207d-7597-4ebd-ad24-913b6564f5ad\" (UID: \"5448207d-7597-4ebd-ad24-913b6564f5ad\") " Nov 03 21:13:02.120564 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:02.120547 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5448207d-7597-4ebd-ad24-913b6564f5ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5448207d-7597-4ebd-ad24-913b6564f5ad-maas-api-openshift-ca\") pod \"5448207d-7597-4ebd-ad24-913b6564f5ad\" (UID: \"5448207d-7597-4ebd-ad24-913b6564f5ad\") " Nov 03 21:13:02.120623 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:02.120595 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5448207d-7597-4ebd-ad24-913b6564f5ad-kube-api-access-22p8b\" (UniqueName: \"kubernetes.io/projected/5448207d-7597-4ebd-ad24-913b6564f5ad-kube-api-access-22p8b\") pod \"5448207d-7597-4ebd-ad24-913b6564f5ad\" (UID: \"5448207d-7597-4ebd-ad24-913b6564f5ad\") " Nov 03 21:13:02.120898 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:02.120873 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5448207d-7597-4ebd-ad24-913b6564f5ad-maas-api-openshift-ca" pod "5448207d-7597-4ebd-ad24-913b6564f5ad" (UID: "5448207d-7597-4ebd-ad24-913b6564f5ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:13:02.121041 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:02.121026 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5448207d-7597-4ebd-ad24-913b6564f5ad-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:02.122576 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:02.122552 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5448207d-7597-4ebd-ad24-913b6564f5ad-maas-api-xks-ca" pod "5448207d-7597-4ebd-ad24-913b6564f5ad" (UID: "5448207d-7597-4ebd-ad24-913b6564f5ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:13:02.122811 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:02.122790 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5448207d-7597-4ebd-ad24-913b6564f5ad-kube-api-access-22p8b" pod "5448207d-7597-4ebd-ad24-913b6564f5ad" (UID: "5448207d-7597-4ebd-ad24-913b6564f5ad"). InnerVolumeSpecName "kube-api-access-22p8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:13:02.222210 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:02.222173 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22p8b\" (UniqueName: \"kubernetes.io/projected/5448207d-7597-4ebd-ad24-913b6564f5ad-kube-api-access-22p8b\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:02.222210 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:02.222209 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5448207d-7597-4ebd-ad24-913b6564f5ad-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:02.846362 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:02.846330 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0944521f-8t75q" Nov 03 21:13:02.846868 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:02.846363 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0944521f-8t75q" event={"ID":"5448207d-7597-4ebd-ad24-913b6564f5ad","Type":"ContainerDied","Data":"d4cdfb821d25bbaeed7b327974111043d45ea9e40fb56f3ef1bd73ae87bab763"} Nov 03 21:13:02.846868 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:02.846413 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4cdfb821d25bbaeed7b327974111043d45ea9e40fb56f3ef1bd73ae87bab763" Nov 03 21:13:02.897242 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:02.897204 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e7f30a9-0862-48a5-9ab7-e692e75bb294" path="/var/lib/kubelet/pods/6e7f30a9-0862-48a5-9ab7-e692e75bb294/volumes" Nov 03 21:13:02.897694 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:02.897678 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aab129fc-dcb8-4cd0-beaa-49f36dfb3f38" path="/var/lib/kubelet/pods/aab129fc-dcb8-4cd0-beaa-49f36dfb3f38/volumes" Nov 03 21:13:02.898073 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:02.898060 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5d3dcaa-a51a-40c6-8049-871b300552e1" path="/var/lib/kubelet/pods/d5d3dcaa-a51a-40c6-8049-871b300552e1/volumes" Nov 03 21:13:02.898777 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:02.898762 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffdb1619-de55-4707-8dbf-b40be40c608e" path="/var/lib/kubelet/pods/ffdb1619-de55-4707-8dbf-b40be40c608e/volumes" Nov 03 21:13:06.939656 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:06.939563 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-32c41e-76q72"] Nov 03 21:13:06.940533 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:06.940512 4305 cpu_manager.go:414] "containerMap: removing container" podUID="aab129fc-dcb8-4cd0-beaa-49f36dfb3f38" containerName="payload-pre-processing" Nov 03 21:13:06.940665 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:06.940534 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aab129fc-dcb8-4cd0-beaa-49f36dfb3f38" containerName="payload-pre-processing" Nov 03 21:13:06.940665 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:06.940553 4305 cpu_manager.go:414] "containerMap: removing container" podUID="d5d3dcaa-a51a-40c6-8049-871b300552e1" containerName="main" Nov 03 21:13:06.940665 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:06.940560 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5d3dcaa-a51a-40c6-8049-871b300552e1" containerName="main" Nov 03 21:13:06.940665 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:06.940574 4305 cpu_manager.go:414] "containerMap: removing container" podUID="8a5f2262-0ce7-4ad7-89f0-a1145d1da1dd" containerName="revoke-keys" Nov 03 21:13:06.940665 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:06.940583 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a5f2262-0ce7-4ad7-89f0-a1145d1da1dd" containerName="revoke-keys" Nov 03 21:13:06.940665 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:06.940610 4305 cpu_manager.go:414] "containerMap: removing container" podUID="6e7f30a9-0862-48a5-9ab7-e692e75bb294" containerName="main" Nov 03 21:13:06.940665 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:06.940618 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e7f30a9-0862-48a5-9ab7-e692e75bb294" containerName="main" Nov 03 21:13:06.940665 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:06.940631 4305 cpu_manager.go:414] "containerMap: removing container" podUID="d6b6e186-36b1-4e38-9ca2-187dc868fc2b" containerName="revoke-keys" Nov 03 21:13:06.940665 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:06.940637 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6b6e186-36b1-4e38-9ca2-187dc868fc2b" containerName="revoke-keys" Nov 03 21:13:06.940665 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:06.940658 4305 cpu_manager.go:414] "containerMap: removing container" podUID="5448207d-7597-4ebd-ad24-913b6564f5ad" containerName="revoke-keys" Nov 03 21:13:06.940665 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:06.940664 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5448207d-7597-4ebd-ad24-913b6564f5ad" containerName="revoke-keys" Nov 03 21:13:06.941283 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:06.940677 4305 cpu_manager.go:414] "containerMap: removing container" podUID="ffdb1619-de55-4707-8dbf-b40be40c608e" containerName="payload-processing" Nov 03 21:13:06.941283 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:06.940684 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffdb1619-de55-4707-8dbf-b40be40c608e" containerName="payload-processing" Nov 03 21:13:06.941283 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:06.940853 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="5448207d-7597-4ebd-ad24-913b6564f5ad" containerName="revoke-keys" Nov 03 21:13:06.941283 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:06.940869 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffdb1619-de55-4707-8dbf-b40be40c608e" containerName="payload-processing" Nov 03 21:13:06.941283 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:06.940886 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="aab129fc-dcb8-4cd0-beaa-49f36dfb3f38" containerName="payload-pre-processing" Nov 03 21:13:06.941283 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:06.940898 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5d3dcaa-a51a-40c6-8049-871b300552e1" containerName="main" Nov 03 21:13:06.941283 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:06.940908 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a5f2262-0ce7-4ad7-89f0-a1145d1da1dd" containerName="revoke-keys" Nov 03 21:13:06.941283 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:06.940920 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e7f30a9-0862-48a5-9ab7-e692e75bb294" containerName="main" Nov 03 21:13:06.941283 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:06.940930 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6b6e186-36b1-4e38-9ca2-187dc868fc2b" containerName="revoke-keys" Nov 03 21:13:06.946093 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:06.946061 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-32c41e-76q72" Nov 03 21:13:07.005682 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:07.005650 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-32c41e-76q72"] Nov 03 21:13:07.050562 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:07.050527 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:13:07.050754 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:07.050610 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:13:07.050754 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:07.050625 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:13:07.050754 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:07.050665 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77vtd\"" Nov 03 21:13:07.093054 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:07.093012 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6324c8df-a6a0-46c7-ae3a-3a99d2eb186e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-32c41e-76q72\" (UID: \"6324c8df-a6a0-46c7-ae3a-3a99d2eb186e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-32c41e-76q72" Nov 03 21:13:07.093054 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:07.093053 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2fnr\" (UniqueName: \"kubernetes.io/projected/6324c8df-a6a0-46c7-ae3a-3a99d2eb186e-kube-api-access-x2fnr\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-32c41e-76q72\" (UID: \"6324c8df-a6a0-46c7-ae3a-3a99d2eb186e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-32c41e-76q72" Nov 03 21:13:07.093289 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:07.093079 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6324c8df-a6a0-46c7-ae3a-3a99d2eb186e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-32c41e-76q72\" (UID: \"6324c8df-a6a0-46c7-ae3a-3a99d2eb186e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-32c41e-76q72" Nov 03 21:13:07.194687 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:07.194602 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6324c8df-a6a0-46c7-ae3a-3a99d2eb186e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-32c41e-76q72\" (UID: \"6324c8df-a6a0-46c7-ae3a-3a99d2eb186e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-32c41e-76q72" Nov 03 21:13:07.194687 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:07.194637 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2fnr\" (UniqueName: \"kubernetes.io/projected/6324c8df-a6a0-46c7-ae3a-3a99d2eb186e-kube-api-access-x2fnr\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-32c41e-76q72\" (UID: \"6324c8df-a6a0-46c7-ae3a-3a99d2eb186e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-32c41e-76q72" Nov 03 21:13:07.194687 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:07.194659 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6324c8df-a6a0-46c7-ae3a-3a99d2eb186e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-32c41e-76q72\" (UID: \"6324c8df-a6a0-46c7-ae3a-3a99d2eb186e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-32c41e-76q72" Nov 03 21:13:07.195592 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:07.195568 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6324c8df-a6a0-46c7-ae3a-3a99d2eb186e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-32c41e-76q72\" (UID: \"6324c8df-a6a0-46c7-ae3a-3a99d2eb186e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-32c41e-76q72" Nov 03 21:13:07.197167 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:07.197143 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6324c8df-a6a0-46c7-ae3a-3a99d2eb186e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-32c41e-76q72\" (UID: \"6324c8df-a6a0-46c7-ae3a-3a99d2eb186e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-32c41e-76q72" Nov 03 21:13:07.258674 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:07.258638 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2fnr\" (UniqueName: \"kubernetes.io/projected/6324c8df-a6a0-46c7-ae3a-3a99d2eb186e-kube-api-access-x2fnr\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-32c41e-76q72\" (UID: \"6324c8df-a6a0-46c7-ae3a-3a99d2eb186e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-32c41e-76q72" Nov 03 21:13:07.259483 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:07.259460 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-32c41e-76q72" Nov 03 21:13:07.640767 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:13:07.640728 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6324c8df_a6a0_46c7_ae3a_3a99d2eb186e.slice/crio-903ae6e1a2a4b1100d8a5190ae8a18567780f7978c5fe57623f70c28803bbd73 WatchSource:0}: Error finding container 903ae6e1a2a4b1100d8a5190ae8a18567780f7978c5fe57623f70c28803bbd73: Status 404 returned error can't find the container with id 903ae6e1a2a4b1100d8a5190ae8a18567780f7978c5fe57623f70c28803bbd73 Nov 03 21:13:07.641332 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:07.641311 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-32c41e-76q72"] Nov 03 21:13:07.885929 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:07.885898 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-32c41e-76q72" event={"ID":"6324c8df-a6a0-46c7-ae3a-3a99d2eb186e","Type":"ContainerStarted","Data":"903ae6e1a2a4b1100d8a5190ae8a18567780f7978c5fe57623f70c28803bbd73"} Nov 03 21:13:08.895678 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:08.895642 4305 generic.go:356] "Generic (PLEG): container finished" podID="6324c8df-a6a0-46c7-ae3a-3a99d2eb186e" containerID="22b644f0ac2be81397cd6fca321a384427619a498e5af71ae61eb84895f43561" exitCode=0 Nov 03 21:13:08.896144 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:08.895834 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-32c41e-76q72" event={"ID":"6324c8df-a6a0-46c7-ae3a-3a99d2eb186e","Type":"ContainerDied","Data":"22b644f0ac2be81397cd6fca321a384427619a498e5af71ae61eb84895f43561"} Nov 03 21:13:09.309954 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:09.309919 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-1e5494-lpmfx"] Nov 03 21:13:09.314147 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:09.314128 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-1e5494-lpmfx" Nov 03 21:13:09.372141 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:09.372104 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-1e5494-lpmfx"] Nov 03 21:13:09.451003 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:09.450956 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mpgl\" (UniqueName: \"kubernetes.io/projected/16af98b0-b553-4289-b356-3d8757f32e4c-kube-api-access-9mpgl\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-1e5494-lpmfx\" (UID: \"16af98b0-b553-4289-b356-3d8757f32e4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-1e5494-lpmfx" Nov 03 21:13:09.451199 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:09.451139 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16af98b0-b553-4289-b356-3d8757f32e4c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-1e5494-lpmfx\" (UID: \"16af98b0-b553-4289-b356-3d8757f32e4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-1e5494-lpmfx" Nov 03 21:13:09.451199 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:09.451168 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16af98b0-b553-4289-b356-3d8757f32e4c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-1e5494-lpmfx\" (UID: \"16af98b0-b553-4289-b356-3d8757f32e4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-1e5494-lpmfx" Nov 03 21:13:09.552729 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:09.552689 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mpgl\" (UniqueName: \"kubernetes.io/projected/16af98b0-b553-4289-b356-3d8757f32e4c-kube-api-access-9mpgl\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-1e5494-lpmfx\" (UID: \"16af98b0-b553-4289-b356-3d8757f32e4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-1e5494-lpmfx" Nov 03 21:13:09.552942 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:09.552827 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16af98b0-b553-4289-b356-3d8757f32e4c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-1e5494-lpmfx\" (UID: \"16af98b0-b553-4289-b356-3d8757f32e4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-1e5494-lpmfx" Nov 03 21:13:09.552942 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:09.552848 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16af98b0-b553-4289-b356-3d8757f32e4c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-1e5494-lpmfx\" (UID: \"16af98b0-b553-4289-b356-3d8757f32e4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-1e5494-lpmfx" Nov 03 21:13:09.553493 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:09.553464 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16af98b0-b553-4289-b356-3d8757f32e4c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-1e5494-lpmfx\" (UID: \"16af98b0-b553-4289-b356-3d8757f32e4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-1e5494-lpmfx" Nov 03 21:13:09.555402 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:09.555381 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16af98b0-b553-4289-b356-3d8757f32e4c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-1e5494-lpmfx\" (UID: \"16af98b0-b553-4289-b356-3d8757f32e4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-1e5494-lpmfx" Nov 03 21:13:09.614372 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:09.614300 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mpgl\" (UniqueName: \"kubernetes.io/projected/16af98b0-b553-4289-b356-3d8757f32e4c-kube-api-access-9mpgl\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-1e5494-lpmfx\" (UID: \"16af98b0-b553-4289-b356-3d8757f32e4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-1e5494-lpmfx" Nov 03 21:13:09.643911 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:09.642694 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-1e5494-lpmfx" Nov 03 21:13:10.111069 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:10.111037 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-32c41e-76q72" Nov 03 21:13:10.178918 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:10.178888 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-1e5494-lpmfx"] Nov 03 21:13:10.179428 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:13:10.179404 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16af98b0_b553_4289_b356_3d8757f32e4c.slice/crio-cb4a52c875c506235c9c846e5e0989f593c7d5729a90d6e519b31d5d39adb05c WatchSource:0}: Error finding container cb4a52c875c506235c9c846e5e0989f593c7d5729a90d6e519b31d5d39adb05c: Status 404 returned error can't find the container with id cb4a52c875c506235c9c846e5e0989f593c7d5729a90d6e519b31d5d39adb05c Nov 03 21:13:10.219836 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:10.219805 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6324c8df-a6a0-46c7-ae3a-3a99d2eb186e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6324c8df-a6a0-46c7-ae3a-3a99d2eb186e-maas-api-xks-ca\") pod \"6324c8df-a6a0-46c7-ae3a-3a99d2eb186e\" (UID: \"6324c8df-a6a0-46c7-ae3a-3a99d2eb186e\") " Nov 03 21:13:10.220008 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:10.219930 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6324c8df-a6a0-46c7-ae3a-3a99d2eb186e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6324c8df-a6a0-46c7-ae3a-3a99d2eb186e-maas-api-openshift-ca\") pod \"6324c8df-a6a0-46c7-ae3a-3a99d2eb186e\" (UID: \"6324c8df-a6a0-46c7-ae3a-3a99d2eb186e\") " Nov 03 21:13:10.220106 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:10.220084 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6324c8df-a6a0-46c7-ae3a-3a99d2eb186e-kube-api-access-x2fnr\" (UniqueName: \"kubernetes.io/projected/6324c8df-a6a0-46c7-ae3a-3a99d2eb186e-kube-api-access-x2fnr\") pod \"6324c8df-a6a0-46c7-ae3a-3a99d2eb186e\" (UID: \"6324c8df-a6a0-46c7-ae3a-3a99d2eb186e\") " Nov 03 21:13:10.220309 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:10.220283 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6324c8df-a6a0-46c7-ae3a-3a99d2eb186e-maas-api-openshift-ca" pod "6324c8df-a6a0-46c7-ae3a-3a99d2eb186e" (UID: "6324c8df-a6a0-46c7-ae3a-3a99d2eb186e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:13:10.220665 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:10.220643 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6324c8df-a6a0-46c7-ae3a-3a99d2eb186e-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:10.222245 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:10.222222 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6324c8df-a6a0-46c7-ae3a-3a99d2eb186e-maas-api-xks-ca" pod "6324c8df-a6a0-46c7-ae3a-3a99d2eb186e" (UID: "6324c8df-a6a0-46c7-ae3a-3a99d2eb186e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:13:10.222324 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:10.222254 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6324c8df-a6a0-46c7-ae3a-3a99d2eb186e-kube-api-access-x2fnr" pod "6324c8df-a6a0-46c7-ae3a-3a99d2eb186e" (UID: "6324c8df-a6a0-46c7-ae3a-3a99d2eb186e"). InnerVolumeSpecName "kube-api-access-x2fnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:13:10.322691 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:10.322647 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2fnr\" (UniqueName: \"kubernetes.io/projected/6324c8df-a6a0-46c7-ae3a-3a99d2eb186e-kube-api-access-x2fnr\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:10.322691 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:10.322693 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6324c8df-a6a0-46c7-ae3a-3a99d2eb186e-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:10.924278 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:10.924239 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-32c41e-76q72" event={"ID":"6324c8df-a6a0-46c7-ae3a-3a99d2eb186e","Type":"ContainerDied","Data":"903ae6e1a2a4b1100d8a5190ae8a18567780f7978c5fe57623f70c28803bbd73"} Nov 03 21:13:10.924466 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:10.924296 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="903ae6e1a2a4b1100d8a5190ae8a18567780f7978c5fe57623f70c28803bbd73" Nov 03 21:13:10.924466 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:10.924252 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-32c41e-76q72" Nov 03 21:13:10.926117 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:10.926089 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-1e5494-lpmfx" event={"ID":"16af98b0-b553-4289-b356-3d8757f32e4c","Type":"ContainerStarted","Data":"69651833ddc73ae7bcbe01903d98052cb0cb1248d93bc00d6688e69ae789248b"} Nov 03 21:13:10.926256 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:10.926136 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-1e5494-lpmfx" event={"ID":"16af98b0-b553-4289-b356-3d8757f32e4c","Type":"ContainerStarted","Data":"cb4a52c875c506235c9c846e5e0989f593c7d5729a90d6e519b31d5d39adb05c"} Nov 03 21:13:11.046567 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:11.046515 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-1e5494-lpmfx" podStartSLOduration=2.046500197 podStartE2EDuration="2.046500197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:13:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:13:11.042763247 +0000 UTC m=+2026.268083039" watchObservedRunningTime="2026-11-03 21:13:11.046500197 +0000 UTC m=+2026.271819988" Nov 03 21:13:11.936367 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:11.936331 4305 generic.go:356] "Generic (PLEG): container finished" podID="16af98b0-b553-4289-b356-3d8757f32e4c" containerID="69651833ddc73ae7bcbe01903d98052cb0cb1248d93bc00d6688e69ae789248b" exitCode=0 Nov 03 21:13:11.936806 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:11.936414 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-1e5494-lpmfx" event={"ID":"16af98b0-b553-4289-b356-3d8757f32e4c","Type":"ContainerDied","Data":"69651833ddc73ae7bcbe01903d98052cb0cb1248d93bc00d6688e69ae789248b"} Nov 03 21:13:13.123401 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:13.123374 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-1e5494-lpmfx" Nov 03 21:13:13.160408 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:13.160370 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16af98b0-b553-4289-b356-3d8757f32e4c-kube-api-access-9mpgl\" (UniqueName: \"kubernetes.io/projected/16af98b0-b553-4289-b356-3d8757f32e4c-kube-api-access-9mpgl\") pod \"16af98b0-b553-4289-b356-3d8757f32e4c\" (UID: \"16af98b0-b553-4289-b356-3d8757f32e4c\") " Nov 03 21:13:13.160638 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:13.160577 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16af98b0-b553-4289-b356-3d8757f32e4c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16af98b0-b553-4289-b356-3d8757f32e4c-maas-api-xks-ca\") pod \"16af98b0-b553-4289-b356-3d8757f32e4c\" (UID: \"16af98b0-b553-4289-b356-3d8757f32e4c\") " Nov 03 21:13:13.160735 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:13.160640 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16af98b0-b553-4289-b356-3d8757f32e4c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16af98b0-b553-4289-b356-3d8757f32e4c-maas-api-openshift-ca\") pod \"16af98b0-b553-4289-b356-3d8757f32e4c\" (UID: \"16af98b0-b553-4289-b356-3d8757f32e4c\") " Nov 03 21:13:13.161029 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:13.160998 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16af98b0-b553-4289-b356-3d8757f32e4c-maas-api-openshift-ca" pod "16af98b0-b553-4289-b356-3d8757f32e4c" (UID: "16af98b0-b553-4289-b356-3d8757f32e4c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:13:13.162701 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:13.162676 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16af98b0-b553-4289-b356-3d8757f32e4c-maas-api-xks-ca" pod "16af98b0-b553-4289-b356-3d8757f32e4c" (UID: "16af98b0-b553-4289-b356-3d8757f32e4c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:13:13.162840 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:13.162822 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16af98b0-b553-4289-b356-3d8757f32e4c-kube-api-access-9mpgl" pod "16af98b0-b553-4289-b356-3d8757f32e4c" (UID: "16af98b0-b553-4289-b356-3d8757f32e4c"). InnerVolumeSpecName "kube-api-access-9mpgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:13:13.262577 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:13.262490 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16af98b0-b553-4289-b356-3d8757f32e4c-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:13.262577 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:13.262523 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mpgl\" (UniqueName: \"kubernetes.io/projected/16af98b0-b553-4289-b356-3d8757f32e4c-kube-api-access-9mpgl\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:13.262577 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:13.262534 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16af98b0-b553-4289-b356-3d8757f32e4c-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:13.952326 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:13.952288 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-1e5494-lpmfx" Nov 03 21:13:13.952506 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:13.952329 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-1e5494-lpmfx" event={"ID":"16af98b0-b553-4289-b356-3d8757f32e4c","Type":"ContainerDied","Data":"cb4a52c875c506235c9c846e5e0989f593c7d5729a90d6e519b31d5d39adb05c"} Nov 03 21:13:13.952506 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:13.952383 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb4a52c875c506235c9c846e5e0989f593c7d5729a90d6e519b31d5d39adb05c" Nov 03 21:13:16.323683 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:13:16.323641 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:13:17.179614 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:17.175686 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b8ac68-hqlqx"] Nov 03 21:13:17.224750 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:17.224715 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b8ac68-hqlqx"] Nov 03 21:13:17.631015 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:17.630928 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-32c41e-794b58b44-25fvr"] Nov 03 21:13:17.741635 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:17.741598 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-32c41e-57d86546c-26fn2"] Nov 03 21:13:18.895811 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:18.895775 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6b6e186-36b1-4e38-9ca2-187dc868fc2b" path="/var/lib/kubelet/pods/d6b6e186-36b1-4e38-9ca2-187dc868fc2b/volumes" Nov 03 21:13:20.282536 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:20.282501 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-1e5494-7cd9cc6ffd-jqr7r"] Nov 03 21:13:20.402657 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:20.402622 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-1e5494-5599ddf75f-b2h97"] Nov 03 21:13:22.631838 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:22.631792 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-32c41e-794b58b44-25fvr" podUID="733d5cad-b828-421b-98b9-7077e03cf1d7" containerName="payload-processing" containerID="cri-o://66f77f42bbb292c2db6e29066347423f5be97d9b985c98cdbaf9b13764c5e6ea" gracePeriod=25 Nov 03 21:13:22.743018 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:22.742967 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-32c41e-57d86546c-26fn2" podUID="e9c63228-f4d1-4327-9940-eed444b94174" containerName="payload-pre-processing" containerID="cri-o://32cc2dd7c1f7e37dcc4a984af1527eadfbebde5d9f580a2356dfbcc4e58eb5be" gracePeriod=25 Nov 03 21:13:22.928764 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:22.928744 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-32c41e-794b58b44-25fvr" Nov 03 21:13:23.021883 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:23.021849 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/733d5cad-b828-421b-98b9-7077e03cf1d7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/733d5cad-b828-421b-98b9-7077e03cf1d7-plugins-config-volume\") pod \"733d5cad-b828-421b-98b9-7077e03cf1d7\" (UID: \"733d5cad-b828-421b-98b9-7077e03cf1d7\") " Nov 03 21:13:23.022111 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:23.022091 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/733d5cad-b828-421b-98b9-7077e03cf1d7-kube-api-access-hrc5r\" (UniqueName: \"kubernetes.io/projected/733d5cad-b828-421b-98b9-7077e03cf1d7-kube-api-access-hrc5r\") pod \"733d5cad-b828-421b-98b9-7077e03cf1d7\" (UID: \"733d5cad-b828-421b-98b9-7077e03cf1d7\") " Nov 03 21:13:23.022258 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:23.022238 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/733d5cad-b828-421b-98b9-7077e03cf1d7-plugins-config-volume" pod "733d5cad-b828-421b-98b9-7077e03cf1d7" (UID: "733d5cad-b828-421b-98b9-7077e03cf1d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:13:23.022551 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:23.022530 4305 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/733d5cad-b828-421b-98b9-7077e03cf1d7-plugins-config-volume\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:23.024403 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:23.024370 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/733d5cad-b828-421b-98b9-7077e03cf1d7-kube-api-access-hrc5r" pod "733d5cad-b828-421b-98b9-7077e03cf1d7" (UID: "733d5cad-b828-421b-98b9-7077e03cf1d7"). InnerVolumeSpecName "kube-api-access-hrc5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:13:23.026064 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:23.026040 4305 generic.go:356] "Generic (PLEG): container finished" podID="e9c63228-f4d1-4327-9940-eed444b94174" containerID="32cc2dd7c1f7e37dcc4a984af1527eadfbebde5d9f580a2356dfbcc4e58eb5be" exitCode=0 Nov 03 21:13:23.026153 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:23.026113 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-32c41e-57d86546c-26fn2" event={"ID":"e9c63228-f4d1-4327-9940-eed444b94174","Type":"ContainerDied","Data":"32cc2dd7c1f7e37dcc4a984af1527eadfbebde5d9f580a2356dfbcc4e58eb5be"} Nov 03 21:13:23.027628 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:23.027605 4305 generic.go:356] "Generic (PLEG): container finished" podID="733d5cad-b828-421b-98b9-7077e03cf1d7" containerID="66f77f42bbb292c2db6e29066347423f5be97d9b985c98cdbaf9b13764c5e6ea" exitCode=0 Nov 03 21:13:23.027738 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:23.027654 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-32c41e-794b58b44-25fvr" Nov 03 21:13:23.027738 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:23.027687 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-32c41e-794b58b44-25fvr" event={"ID":"733d5cad-b828-421b-98b9-7077e03cf1d7","Type":"ContainerDied","Data":"66f77f42bbb292c2db6e29066347423f5be97d9b985c98cdbaf9b13764c5e6ea"} Nov 03 21:13:23.027836 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:23.027743 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-32c41e-794b58b44-25fvr" event={"ID":"733d5cad-b828-421b-98b9-7077e03cf1d7","Type":"ContainerDied","Data":"c1c29dab9d0e7ee050ec99c5447eb27e3d23a44c3d37b67cedc0376f1c70e89a"} Nov 03 21:13:23.027836 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:23.027771 4305 scope.go:122] "RemoveContainer" containerID="66f77f42bbb292c2db6e29066347423f5be97d9b985c98cdbaf9b13764c5e6ea" Nov 03 21:13:23.040745 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:23.040727 4305 scope.go:122] "RemoveContainer" containerID="66f77f42bbb292c2db6e29066347423f5be97d9b985c98cdbaf9b13764c5e6ea" Nov 03 21:13:23.041003 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:13:23.040983 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66f77f42bbb292c2db6e29066347423f5be97d9b985c98cdbaf9b13764c5e6ea\": container with ID starting with 66f77f42bbb292c2db6e29066347423f5be97d9b985c98cdbaf9b13764c5e6ea not found: ID does not exist" containerID="66f77f42bbb292c2db6e29066347423f5be97d9b985c98cdbaf9b13764c5e6ea" Nov 03 21:13:23.041069 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:23.041014 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66f77f42bbb292c2db6e29066347423f5be97d9b985c98cdbaf9b13764c5e6ea"} err="failed to get container status \"66f77f42bbb292c2db6e29066347423f5be97d9b985c98cdbaf9b13764c5e6ea\": rpc error: code = NotFound desc = could not find container \"66f77f42bbb292c2db6e29066347423f5be97d9b985c98cdbaf9b13764c5e6ea\": container with ID starting with 66f77f42bbb292c2db6e29066347423f5be97d9b985c98cdbaf9b13764c5e6ea not found: ID does not exist" Nov 03 21:13:23.043515 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:23.043495 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-32c41e-57d86546c-26fn2" Nov 03 21:13:23.123833 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:23.123800 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9c63228-f4d1-4327-9940-eed444b94174-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9c63228-f4d1-4327-9940-eed444b94174-plugins-config-volume\") pod \"e9c63228-f4d1-4327-9940-eed444b94174\" (UID: \"e9c63228-f4d1-4327-9940-eed444b94174\") " Nov 03 21:13:23.124052 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:23.123953 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9c63228-f4d1-4327-9940-eed444b94174-kube-api-access-8js5x\" (UniqueName: \"kubernetes.io/projected/e9c63228-f4d1-4327-9940-eed444b94174-kube-api-access-8js5x\") pod \"e9c63228-f4d1-4327-9940-eed444b94174\" (UID: \"e9c63228-f4d1-4327-9940-eed444b94174\") " Nov 03 21:13:23.124275 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:23.124245 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9c63228-f4d1-4327-9940-eed444b94174-plugins-config-volume" pod "e9c63228-f4d1-4327-9940-eed444b94174" (UID: "e9c63228-f4d1-4327-9940-eed444b94174"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:13:23.124626 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:23.124599 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrc5r\" (UniqueName: \"kubernetes.io/projected/733d5cad-b828-421b-98b9-7077e03cf1d7-kube-api-access-hrc5r\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:23.124626 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:23.124624 4305 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9c63228-f4d1-4327-9940-eed444b94174-plugins-config-volume\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:23.126321 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:23.126299 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9c63228-f4d1-4327-9940-eed444b94174-kube-api-access-8js5x" pod "e9c63228-f4d1-4327-9940-eed444b94174" (UID: "e9c63228-f4d1-4327-9940-eed444b94174"). InnerVolumeSpecName "kube-api-access-8js5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:13:23.226247 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:23.226151 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8js5x\" (UniqueName: \"kubernetes.io/projected/e9c63228-f4d1-4327-9940-eed444b94174-kube-api-access-8js5x\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:23.350315 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:23.350279 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-32c41e-794b58b44-25fvr"] Nov 03 21:13:23.360044 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:23.360018 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-32c41e-794b58b44-25fvr"] Nov 03 21:13:24.036949 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:24.036867 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-32c41e-57d86546c-26fn2" Nov 03 21:13:24.036949 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:24.036921 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-32c41e-57d86546c-26fn2" event={"ID":"e9c63228-f4d1-4327-9940-eed444b94174","Type":"ContainerDied","Data":"ca2f5a69f78d3deb0bab0d557a486d11f3dbed4bda6af2092841d236708fdb7b"} Nov 03 21:13:24.037536 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:24.036981 4305 scope.go:122] "RemoveContainer" containerID="32cc2dd7c1f7e37dcc4a984af1527eadfbebde5d9f580a2356dfbcc4e58eb5be" Nov 03 21:13:24.205958 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:24.205923 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-32c41e-57d86546c-26fn2"] Nov 03 21:13:24.254900 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:24.254865 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-32c41e-57d86546c-26fn2"] Nov 03 21:13:24.896450 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:24.896407 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="733d5cad-b828-421b-98b9-7077e03cf1d7" path="/var/lib/kubelet/pods/733d5cad-b828-421b-98b9-7077e03cf1d7/volumes" Nov 03 21:13:24.896838 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:24.896822 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9c63228-f4d1-4327-9940-eed444b94174" path="/var/lib/kubelet/pods/e9c63228-f4d1-4327-9940-eed444b94174/volumes" Nov 03 21:13:25.283780 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:25.283669 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-1e5494-7cd9cc6ffd-jqr7r" podUID="dc0be43c-23aa-4e0c-b69a-34162e3fa0db" containerName="payload-processing" containerID="cri-o://ef7dd8c890e58de3f9bfe1cbe49ebdde5544ee9562c388bb363c952d9587d353" gracePeriod=25 Nov 03 21:13:25.403396 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:25.403316 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-1e5494-5599ddf75f-b2h97" podUID="5cf1b90d-073b-4cbd-8d75-e23a6ab6590a" containerName="payload-pre-processing" containerID="cri-o://32e968ec56691ac3bbd2dbe3af3f96d0f9d9f7f9defa446a023eacaf634e7968" gracePeriod=25 Nov 03 21:13:25.595648 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:25.595623 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-1e5494-7cd9cc6ffd-jqr7r" Nov 03 21:13:25.709796 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:25.709767 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-1e5494-5599ddf75f-b2h97" Nov 03 21:13:25.761379 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:25.761345 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc0be43c-23aa-4e0c-b69a-34162e3fa0db-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc0be43c-23aa-4e0c-b69a-34162e3fa0db-plugins-config-volume\") pod \"dc0be43c-23aa-4e0c-b69a-34162e3fa0db\" (UID: \"dc0be43c-23aa-4e0c-b69a-34162e3fa0db\") " Nov 03 21:13:25.761567 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:25.761476 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc0be43c-23aa-4e0c-b69a-34162e3fa0db-kube-api-access-hzzqp\" (UniqueName: \"kubernetes.io/projected/dc0be43c-23aa-4e0c-b69a-34162e3fa0db-kube-api-access-hzzqp\") pod \"dc0be43c-23aa-4e0c-b69a-34162e3fa0db\" (UID: \"dc0be43c-23aa-4e0c-b69a-34162e3fa0db\") " Nov 03 21:13:25.761731 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:25.761707 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc0be43c-23aa-4e0c-b69a-34162e3fa0db-plugins-config-volume" pod "dc0be43c-23aa-4e0c-b69a-34162e3fa0db" (UID: "dc0be43c-23aa-4e0c-b69a-34162e3fa0db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:13:25.762128 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:25.762106 4305 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc0be43c-23aa-4e0c-b69a-34162e3fa0db-plugins-config-volume\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:25.763834 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:25.763812 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc0be43c-23aa-4e0c-b69a-34162e3fa0db-kube-api-access-hzzqp" pod "dc0be43c-23aa-4e0c-b69a-34162e3fa0db" (UID: "dc0be43c-23aa-4e0c-b69a-34162e3fa0db"). InnerVolumeSpecName "kube-api-access-hzzqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:13:25.930735 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:25.930689 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cf1b90d-073b-4cbd-8d75-e23a6ab6590a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cf1b90d-073b-4cbd-8d75-e23a6ab6590a-plugins-config-volume\") pod \"5cf1b90d-073b-4cbd-8d75-e23a6ab6590a\" (UID: \"5cf1b90d-073b-4cbd-8d75-e23a6ab6590a\") " Nov 03 21:13:25.931891 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:25.931870 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cf1b90d-073b-4cbd-8d75-e23a6ab6590a-kube-api-access-mbnzr\" (UniqueName: \"kubernetes.io/projected/5cf1b90d-073b-4cbd-8d75-e23a6ab6590a-kube-api-access-mbnzr\") pod \"5cf1b90d-073b-4cbd-8d75-e23a6ab6590a\" (UID: \"5cf1b90d-073b-4cbd-8d75-e23a6ab6590a\") " Nov 03 21:13:25.932334 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:25.931745 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cf1b90d-073b-4cbd-8d75-e23a6ab6590a-plugins-config-volume" pod "5cf1b90d-073b-4cbd-8d75-e23a6ab6590a" (UID: "5cf1b90d-073b-4cbd-8d75-e23a6ab6590a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:13:25.933051 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:25.933030 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzzqp\" (UniqueName: \"kubernetes.io/projected/dc0be43c-23aa-4e0c-b69a-34162e3fa0db-kube-api-access-hzzqp\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:25.934840 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:25.934820 4305 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cf1b90d-073b-4cbd-8d75-e23a6ab6590a-plugins-config-volume\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:25.962043 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:25.957818 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cf1b90d-073b-4cbd-8d75-e23a6ab6590a-kube-api-access-mbnzr" pod "5cf1b90d-073b-4cbd-8d75-e23a6ab6590a" (UID: "5cf1b90d-073b-4cbd-8d75-e23a6ab6590a"). InnerVolumeSpecName "kube-api-access-mbnzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:13:26.108503 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:26.045774 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbnzr\" (UniqueName: \"kubernetes.io/projected/5cf1b90d-073b-4cbd-8d75-e23a6ab6590a-kube-api-access-mbnzr\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:26.133854 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:26.133812 4305 generic.go:356] "Generic (PLEG): container finished" podID="5cf1b90d-073b-4cbd-8d75-e23a6ab6590a" containerID="32e968ec56691ac3bbd2dbe3af3f96d0f9d9f7f9defa446a023eacaf634e7968" exitCode=0 Nov 03 21:13:26.134032 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:26.133939 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-1e5494-5599ddf75f-b2h97" event={"ID":"5cf1b90d-073b-4cbd-8d75-e23a6ab6590a","Type":"ContainerDied","Data":"32e968ec56691ac3bbd2dbe3af3f96d0f9d9f7f9defa446a023eacaf634e7968"} Nov 03 21:13:26.134032 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:26.133992 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-1e5494-5599ddf75f-b2h97" event={"ID":"5cf1b90d-073b-4cbd-8d75-e23a6ab6590a","Type":"ContainerDied","Data":"a9796911ccd9bb83300c87308df98592b5b8559d450ac2543ff79f805648e290"} Nov 03 21:13:26.134032 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:26.134019 4305 scope.go:122] "RemoveContainer" containerID="32e968ec56691ac3bbd2dbe3af3f96d0f9d9f7f9defa446a023eacaf634e7968" Nov 03 21:13:26.134299 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:26.134180 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-1e5494-5599ddf75f-b2h97" Nov 03 21:13:26.138558 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:26.138530 4305 generic.go:356] "Generic (PLEG): container finished" podID="dc0be43c-23aa-4e0c-b69a-34162e3fa0db" containerID="ef7dd8c890e58de3f9bfe1cbe49ebdde5544ee9562c388bb363c952d9587d353" exitCode=0 Nov 03 21:13:26.139072 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:26.138788 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-1e5494-7cd9cc6ffd-jqr7r" Nov 03 21:13:26.139973 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:26.138822 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-1e5494-7cd9cc6ffd-jqr7r" event={"ID":"dc0be43c-23aa-4e0c-b69a-34162e3fa0db","Type":"ContainerDied","Data":"ef7dd8c890e58de3f9bfe1cbe49ebdde5544ee9562c388bb363c952d9587d353"} Nov 03 21:13:26.140070 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:26.140024 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-1e5494-7cd9cc6ffd-jqr7r" event={"ID":"dc0be43c-23aa-4e0c-b69a-34162e3fa0db","Type":"ContainerDied","Data":"43efc4e022c0589640d0f2363135584a2cc01592c044ca0e2af8fc75c494acdd"} Nov 03 21:13:26.156020 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:26.155909 4305 scope.go:122] "RemoveContainer" containerID="32e968ec56691ac3bbd2dbe3af3f96d0f9d9f7f9defa446a023eacaf634e7968" Nov 03 21:13:26.156337 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:13:26.156314 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32e968ec56691ac3bbd2dbe3af3f96d0f9d9f7f9defa446a023eacaf634e7968\": container with ID starting with 32e968ec56691ac3bbd2dbe3af3f96d0f9d9f7f9defa446a023eacaf634e7968 not found: ID does not exist" containerID="32e968ec56691ac3bbd2dbe3af3f96d0f9d9f7f9defa446a023eacaf634e7968" Nov 03 21:13:26.156465 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:26.156348 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32e968ec56691ac3bbd2dbe3af3f96d0f9d9f7f9defa446a023eacaf634e7968"} err="failed to get container status \"32e968ec56691ac3bbd2dbe3af3f96d0f9d9f7f9defa446a023eacaf634e7968\": rpc error: code = NotFound desc = could not find container \"32e968ec56691ac3bbd2dbe3af3f96d0f9d9f7f9defa446a023eacaf634e7968\": container with ID starting with 32e968ec56691ac3bbd2dbe3af3f96d0f9d9f7f9defa446a023eacaf634e7968 not found: ID does not exist" Nov 03 21:13:26.156465 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:26.156365 4305 scope.go:122] "RemoveContainer" containerID="ef7dd8c890e58de3f9bfe1cbe49ebdde5544ee9562c388bb363c952d9587d353" Nov 03 21:13:26.167043 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:26.167021 4305 scope.go:122] "RemoveContainer" containerID="ef7dd8c890e58de3f9bfe1cbe49ebdde5544ee9562c388bb363c952d9587d353" Nov 03 21:13:26.167395 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:13:26.167372 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef7dd8c890e58de3f9bfe1cbe49ebdde5544ee9562c388bb363c952d9587d353\": container with ID starting with ef7dd8c890e58de3f9bfe1cbe49ebdde5544ee9562c388bb363c952d9587d353 not found: ID does not exist" containerID="ef7dd8c890e58de3f9bfe1cbe49ebdde5544ee9562c388bb363c952d9587d353" Nov 03 21:13:26.167493 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:26.167406 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef7dd8c890e58de3f9bfe1cbe49ebdde5544ee9562c388bb363c952d9587d353"} err="failed to get container status \"ef7dd8c890e58de3f9bfe1cbe49ebdde5544ee9562c388bb363c952d9587d353\": rpc error: code = NotFound desc = could not find container \"ef7dd8c890e58de3f9bfe1cbe49ebdde5544ee9562c388bb363c952d9587d353\": container with ID starting with ef7dd8c890e58de3f9bfe1cbe49ebdde5544ee9562c388bb363c952d9587d353 not found: ID does not exist" Nov 03 21:13:26.308607 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:26.308576 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-1e5494-5599ddf75f-b2h97"] Nov 03 21:13:26.312069 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:26.312040 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-1e5494-5599ddf75f-b2h97"] Nov 03 21:13:26.476147 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:26.476112 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-1e5494-7cd9cc6ffd-jqr7r"] Nov 03 21:13:26.481564 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:26.481537 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-1e5494-7cd9cc6ffd-jqr7r"] Nov 03 21:13:26.897657 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:26.897626 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cf1b90d-073b-4cbd-8d75-e23a6ab6590a" path="/var/lib/kubelet/pods/5cf1b90d-073b-4cbd-8d75-e23a6ab6590a/volumes" Nov 03 21:13:26.898088 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:26.898075 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc0be43c-23aa-4e0c-b69a-34162e3fa0db" path="/var/lib/kubelet/pods/dc0be43c-23aa-4e0c-b69a-34162e3fa0db/volumes" Nov 03 21:13:31.788005 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:31.787973 4305 scope.go:122] "RemoveContainer" containerID="dda29973d6a71c376a65089459088e34aed649e321c091853162af37b5a64abf" Nov 03 21:13:37.538758 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:37.538716 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-01c7a3-5x77g"] Nov 03 21:13:37.539348 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:37.539333 4305 cpu_manager.go:414] "containerMap: removing container" podUID="16af98b0-b553-4289-b356-3d8757f32e4c" containerName="revoke-keys" Nov 03 21:13:37.539398 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:37.539349 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16af98b0-b553-4289-b356-3d8757f32e4c" containerName="revoke-keys" Nov 03 21:13:37.539398 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:37.539362 4305 cpu_manager.go:414] "containerMap: removing container" podUID="dc0be43c-23aa-4e0c-b69a-34162e3fa0db" containerName="payload-processing" Nov 03 21:13:37.539398 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:37.539367 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc0be43c-23aa-4e0c-b69a-34162e3fa0db" containerName="payload-processing" Nov 03 21:13:37.539398 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:37.539397 4305 cpu_manager.go:414] "containerMap: removing container" podUID="5cf1b90d-073b-4cbd-8d75-e23a6ab6590a" containerName="payload-pre-processing" Nov 03 21:13:37.539567 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:37.539403 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cf1b90d-073b-4cbd-8d75-e23a6ab6590a" containerName="payload-pre-processing" Nov 03 21:13:37.539567 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:37.539421 4305 cpu_manager.go:414] "containerMap: removing container" podUID="733d5cad-b828-421b-98b9-7077e03cf1d7" containerName="payload-processing" Nov 03 21:13:37.539567 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:37.539428 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="733d5cad-b828-421b-98b9-7077e03cf1d7" containerName="payload-processing" Nov 03 21:13:37.539567 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:37.539449 4305 cpu_manager.go:414] "containerMap: removing container" podUID="6324c8df-a6a0-46c7-ae3a-3a99d2eb186e" containerName="revoke-keys" Nov 03 21:13:37.539567 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:37.539454 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6324c8df-a6a0-46c7-ae3a-3a99d2eb186e" containerName="revoke-keys" Nov 03 21:13:37.539567 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:37.539465 4305 cpu_manager.go:414] "containerMap: removing container" podUID="e9c63228-f4d1-4327-9940-eed444b94174" containerName="payload-pre-processing" Nov 03 21:13:37.539567 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:37.539470 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9c63228-f4d1-4327-9940-eed444b94174" containerName="payload-pre-processing" Nov 03 21:13:37.539567 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:37.539549 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="733d5cad-b828-421b-98b9-7077e03cf1d7" containerName="payload-processing" Nov 03 21:13:37.539567 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:37.539560 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="6324c8df-a6a0-46c7-ae3a-3a99d2eb186e" containerName="revoke-keys" Nov 03 21:13:37.539841 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:37.539574 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9c63228-f4d1-4327-9940-eed444b94174" containerName="payload-pre-processing" Nov 03 21:13:37.539841 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:37.539580 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="16af98b0-b553-4289-b356-3d8757f32e4c" containerName="revoke-keys" Nov 03 21:13:37.539841 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:37.539588 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc0be43c-23aa-4e0c-b69a-34162e3fa0db" containerName="payload-processing" Nov 03 21:13:37.539841 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:37.539597 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cf1b90d-073b-4cbd-8d75-e23a6ab6590a" containerName="payload-pre-processing" Nov 03 21:13:37.545130 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:37.545107 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-01c7a3-5x77g" Nov 03 21:13:37.595799 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:37.595760 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bf47f9f-ebc7-49ea-ae69-74712e627abb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-01c7a3-5x77g\" (UID: \"7bf47f9f-ebc7-49ea-ae69-74712e627abb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-01c7a3-5x77g" Nov 03 21:13:37.595967 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:37.595813 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxx2r\" (UniqueName: \"kubernetes.io/projected/7bf47f9f-ebc7-49ea-ae69-74712e627abb-kube-api-access-hxx2r\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-01c7a3-5x77g\" (UID: \"7bf47f9f-ebc7-49ea-ae69-74712e627abb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-01c7a3-5x77g" Nov 03 21:13:37.596030 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:37.596002 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bf47f9f-ebc7-49ea-ae69-74712e627abb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-01c7a3-5x77g\" (UID: \"7bf47f9f-ebc7-49ea-ae69-74712e627abb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-01c7a3-5x77g" Nov 03 21:13:37.599694 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:37.599663 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:13:37.647143 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:37.647103 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:13:37.647143 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:37.647108 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77vtd\"" Nov 03 21:13:37.647143 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:37.647108 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:13:37.651051 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:37.651028 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-01c7a3-5x77g"] Nov 03 21:13:37.711160 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:37.711117 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bf47f9f-ebc7-49ea-ae69-74712e627abb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-01c7a3-5x77g\" (UID: \"7bf47f9f-ebc7-49ea-ae69-74712e627abb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-01c7a3-5x77g" Nov 03 21:13:37.714798 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:37.714556 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bf47f9f-ebc7-49ea-ae69-74712e627abb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-01c7a3-5x77g\" (UID: \"7bf47f9f-ebc7-49ea-ae69-74712e627abb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-01c7a3-5x77g" Nov 03 21:13:37.714798 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:37.714634 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxx2r\" (UniqueName: \"kubernetes.io/projected/7bf47f9f-ebc7-49ea-ae69-74712e627abb-kube-api-access-hxx2r\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-01c7a3-5x77g\" (UID: \"7bf47f9f-ebc7-49ea-ae69-74712e627abb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-01c7a3-5x77g" Nov 03 21:13:37.716356 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:37.716327 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bf47f9f-ebc7-49ea-ae69-74712e627abb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-01c7a3-5x77g\" (UID: \"7bf47f9f-ebc7-49ea-ae69-74712e627abb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-01c7a3-5x77g" Nov 03 21:13:37.719995 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:37.719967 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bf47f9f-ebc7-49ea-ae69-74712e627abb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-01c7a3-5x77g\" (UID: \"7bf47f9f-ebc7-49ea-ae69-74712e627abb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-01c7a3-5x77g" Nov 03 21:13:37.775734 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:37.775695 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxx2r\" (UniqueName: \"kubernetes.io/projected/7bf47f9f-ebc7-49ea-ae69-74712e627abb-kube-api-access-hxx2r\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-01c7a3-5x77g\" (UID: \"7bf47f9f-ebc7-49ea-ae69-74712e627abb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-01c7a3-5x77g" Nov 03 21:13:37.859304 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:37.859223 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-01c7a3-5x77g" Nov 03 21:13:38.246883 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:38.246844 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-01c7a3-5x77g"] Nov 03 21:13:38.248317 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:13:38.248289 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bf47f9f_ebc7_49ea_ae69_74712e627abb.slice/crio-11fa12d92c45c13b88c922c98e7985c89e35e6d9014ea6fa9ccd2a09fa21d084 WatchSource:0}: Error finding container 11fa12d92c45c13b88c922c98e7985c89e35e6d9014ea6fa9ccd2a09fa21d084: Status 404 returned error can't find the container with id 11fa12d92c45c13b88c922c98e7985c89e35e6d9014ea6fa9ccd2a09fa21d084 Nov 03 21:13:38.266798 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:38.266772 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-01c7a3-5x77g" event={"ID":"7bf47f9f-ebc7-49ea-ae69-74712e627abb","Type":"ContainerStarted","Data":"11fa12d92c45c13b88c922c98e7985c89e35e6d9014ea6fa9ccd2a09fa21d084"} Nov 03 21:13:38.605857 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:38.605745 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d01cef-ccrrf"] Nov 03 21:13:38.634853 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:38.634818 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d01cef-ccrrf"] Nov 03 21:13:38.896469 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:38.896284 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1e68e0d-aaf7-4c44-8419-5e459ebcfc1c" path="/var/lib/kubelet/pods/a1e68e0d-aaf7-4c44-8419-5e459ebcfc1c/volumes" Nov 03 21:13:39.275172 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:39.275138 4305 generic.go:356] "Generic (PLEG): container finished" podID="7bf47f9f-ebc7-49ea-ae69-74712e627abb" containerID="e8ae5aed3065eceee49db48d8fe2bb21e9fc417af7c418d8912df1c7d3d00c4a" exitCode=0 Nov 03 21:13:39.275343 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:39.275215 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-01c7a3-5x77g" event={"ID":"7bf47f9f-ebc7-49ea-ae69-74712e627abb","Type":"ContainerDied","Data":"e8ae5aed3065eceee49db48d8fe2bb21e9fc417af7c418d8912df1c7d3d00c4a"} Nov 03 21:13:39.734995 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:39.734954 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-32c41e-76q72"] Nov 03 21:13:39.738199 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:39.738168 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-32c41e-76q72"] Nov 03 21:13:40.438002 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:40.437975 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-01c7a3-5x77g" Nov 03 21:13:40.585317 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:40.585197 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bf47f9f-ebc7-49ea-ae69-74712e627abb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bf47f9f-ebc7-49ea-ae69-74712e627abb-maas-api-openshift-ca\") pod \"7bf47f9f-ebc7-49ea-ae69-74712e627abb\" (UID: \"7bf47f9f-ebc7-49ea-ae69-74712e627abb\") " Nov 03 21:13:40.585317 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:40.585314 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bf47f9f-ebc7-49ea-ae69-74712e627abb-kube-api-access-hxx2r\" (UniqueName: \"kubernetes.io/projected/7bf47f9f-ebc7-49ea-ae69-74712e627abb-kube-api-access-hxx2r\") pod \"7bf47f9f-ebc7-49ea-ae69-74712e627abb\" (UID: \"7bf47f9f-ebc7-49ea-ae69-74712e627abb\") " Nov 03 21:13:40.585670 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:40.585360 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bf47f9f-ebc7-49ea-ae69-74712e627abb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bf47f9f-ebc7-49ea-ae69-74712e627abb-maas-api-xks-ca\") pod \"7bf47f9f-ebc7-49ea-ae69-74712e627abb\" (UID: \"7bf47f9f-ebc7-49ea-ae69-74712e627abb\") " Nov 03 21:13:40.587693 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:40.587660 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bf47f9f-ebc7-49ea-ae69-74712e627abb-maas-api-openshift-ca" pod "7bf47f9f-ebc7-49ea-ae69-74712e627abb" (UID: "7bf47f9f-ebc7-49ea-ae69-74712e627abb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:13:40.589360 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:40.589329 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bf47f9f-ebc7-49ea-ae69-74712e627abb-maas-api-xks-ca" pod "7bf47f9f-ebc7-49ea-ae69-74712e627abb" (UID: "7bf47f9f-ebc7-49ea-ae69-74712e627abb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:13:40.591113 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:40.591083 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bf47f9f-ebc7-49ea-ae69-74712e627abb-kube-api-access-hxx2r" pod "7bf47f9f-ebc7-49ea-ae69-74712e627abb" (UID: "7bf47f9f-ebc7-49ea-ae69-74712e627abb"). InnerVolumeSpecName "kube-api-access-hxx2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:13:40.687844 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:40.687799 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bf47f9f-ebc7-49ea-ae69-74712e627abb-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:40.687844 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:40.687833 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxx2r\" (UniqueName: \"kubernetes.io/projected/7bf47f9f-ebc7-49ea-ae69-74712e627abb-kube-api-access-hxx2r\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:40.687844 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:40.687849 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bf47f9f-ebc7-49ea-ae69-74712e627abb-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:40.895649 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:40.895617 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6324c8df-a6a0-46c7-ae3a-3a99d2eb186e" path="/var/lib/kubelet/pods/6324c8df-a6a0-46c7-ae3a-3a99d2eb186e/volumes" Nov 03 21:13:41.291426 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:41.291347 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-01c7a3-5x77g" Nov 03 21:13:41.291426 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:41.291354 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-01c7a3-5x77g" event={"ID":"7bf47f9f-ebc7-49ea-ae69-74712e627abb","Type":"ContainerDied","Data":"11fa12d92c45c13b88c922c98e7985c89e35e6d9014ea6fa9ccd2a09fa21d084"} Nov 03 21:13:41.291426 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:41.291404 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11fa12d92c45c13b88c922c98e7985c89e35e6d9014ea6fa9ccd2a09fa21d084" Nov 03 21:13:41.331565 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:41.331529 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:13:46.041268 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:46.041231 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc"] Nov 03 21:13:46.041774 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:46.041450 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc" podUID="6a908050-5ca3-4977-b279-67965ac30de1" containerName="istio-proxy" containerID="cri-o://fdddee4a24fda9da49e1e23c26953af05b709276af091e6cf4da8eb05fe626b9" gracePeriod=30 Nov 03 21:13:46.055958 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:46.055925 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p"] Nov 03 21:13:46.056186 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:46.056148 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p" podUID="e4b14774-55f2-4fad-88ad-74d08bf328f7" containerName="istio-proxy" containerID="cri-o://2abb835f949043e4a77b448f621626fa8dd5fca6fc7a6fd8ddc5da199fc88179" gracePeriod=30 Nov 03 21:13:46.115575 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:46.115414 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-1e5494-lpmfx"] Nov 03 21:13:46.160898 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:46.153478 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-1e5494-lpmfx"] Nov 03 21:13:46.283331 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:46.283291 4305 patch_prober.go:29] interesting pod/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:13:46.283588 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:46.283355 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p" podUID="e4b14774-55f2-4fad-88ad-74d08bf328f7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:13:46.323880 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:13:46.323787 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:13:46.897473 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:46.897403 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16af98b0-b553-4289-b356-3d8757f32e4c" path="/var/lib/kubelet/pods/16af98b0-b553-4289-b356-3d8757f32e4c/volumes" Nov 03 21:13:47.250052 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:47.249961 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr"] Nov 03 21:13:47.250584 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:47.250228 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr" podUID="953874ed-6376-4775-b514-b09ecb325824" containerName="istio-proxy" containerID="cri-o://3e9b081e86ee58626087431ea3a50f9d3b1d0fdd64224897f303ba82806b26ff" gracePeriod=30 Nov 03 21:13:48.308735 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:48.308698 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e22abf-hncdr"] Nov 03 21:13:48.309298 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:48.309284 4305 cpu_manager.go:414] "containerMap: removing container" podUID="7bf47f9f-ebc7-49ea-ae69-74712e627abb" containerName="revoke-keys" Nov 03 21:13:48.309343 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:48.309300 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bf47f9f-ebc7-49ea-ae69-74712e627abb" containerName="revoke-keys" Nov 03 21:13:48.309393 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:48.309383 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bf47f9f-ebc7-49ea-ae69-74712e627abb" containerName="revoke-keys" Nov 03 21:13:48.312711 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:48.312695 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e22abf-hncdr" Nov 03 21:13:48.372313 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:48.372279 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e22abf-hncdr"] Nov 03 21:13:48.405099 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:48.405060 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da6b7530-9a01-4bfb-bf70-dff574ef1f0f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-e22abf-hncdr\" (UID: \"da6b7530-9a01-4bfb-bf70-dff574ef1f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e22abf-hncdr" Nov 03 21:13:48.405404 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:48.405386 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwd9w\" (UniqueName: \"kubernetes.io/projected/da6b7530-9a01-4bfb-bf70-dff574ef1f0f-kube-api-access-zwd9w\") pod \"maas-api-revoke-keys-e2e-worker-w5-e22abf-hncdr\" (UID: \"da6b7530-9a01-4bfb-bf70-dff574ef1f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e22abf-hncdr" Nov 03 21:13:48.405644 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:48.405619 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da6b7530-9a01-4bfb-bf70-dff574ef1f0f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-e22abf-hncdr\" (UID: \"da6b7530-9a01-4bfb-bf70-dff574ef1f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e22abf-hncdr" Nov 03 21:13:48.417178 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:48.417145 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:13:48.417302 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:48.417178 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77vtd\"" Nov 03 21:13:48.417302 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:48.417178 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:13:48.417302 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:48.417226 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:13:48.506582 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:48.506545 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da6b7530-9a01-4bfb-bf70-dff574ef1f0f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-e22abf-hncdr\" (UID: \"da6b7530-9a01-4bfb-bf70-dff574ef1f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e22abf-hncdr" Nov 03 21:13:48.506810 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:48.506590 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwd9w\" (UniqueName: \"kubernetes.io/projected/da6b7530-9a01-4bfb-bf70-dff574ef1f0f-kube-api-access-zwd9w\") pod \"maas-api-revoke-keys-e2e-worker-w5-e22abf-hncdr\" (UID: \"da6b7530-9a01-4bfb-bf70-dff574ef1f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e22abf-hncdr" Nov 03 21:13:48.506810 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:48.506640 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da6b7530-9a01-4bfb-bf70-dff574ef1f0f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-e22abf-hncdr\" (UID: \"da6b7530-9a01-4bfb-bf70-dff574ef1f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e22abf-hncdr" Nov 03 21:13:48.507313 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:48.507288 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da6b7530-9a01-4bfb-bf70-dff574ef1f0f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-e22abf-hncdr\" (UID: \"da6b7530-9a01-4bfb-bf70-dff574ef1f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e22abf-hncdr" Nov 03 21:13:48.509183 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:48.509163 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da6b7530-9a01-4bfb-bf70-dff574ef1f0f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-e22abf-hncdr\" (UID: \"da6b7530-9a01-4bfb-bf70-dff574ef1f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e22abf-hncdr" Nov 03 21:13:48.567765 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:48.567681 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwd9w\" (UniqueName: \"kubernetes.io/projected/da6b7530-9a01-4bfb-bf70-dff574ef1f0f-kube-api-access-zwd9w\") pod \"maas-api-revoke-keys-e2e-worker-w5-e22abf-hncdr\" (UID: \"da6b7530-9a01-4bfb-bf70-dff574ef1f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e22abf-hncdr" Nov 03 21:13:48.626376 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:48.626345 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e22abf-hncdr" Nov 03 21:13:49.010124 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:49.010095 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e22abf-hncdr"] Nov 03 21:13:49.011751 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:13:49.011722 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda6b7530_9a01_4bfb_bf70_dff574ef1f0f.slice/crio-8629824dc012b337d159fcea59a15025ba550769df9004642f10d51cf52d92ec WatchSource:0}: Error finding container 8629824dc012b337d159fcea59a15025ba550769df9004642f10d51cf52d92ec: Status 404 returned error can't find the container with id 8629824dc012b337d159fcea59a15025ba550769df9004642f10d51cf52d92ec Nov 03 21:13:49.415494 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:49.415457 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e22abf-hncdr" event={"ID":"da6b7530-9a01-4bfb-bf70-dff574ef1f0f","Type":"ContainerStarted","Data":"d7f0175080aa64f5f922196e3f06e75d7c3d0cefc1992fc3fdc1859cf2744cf1"} Nov 03 21:13:49.415932 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:49.415510 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e22abf-hncdr" event={"ID":"da6b7530-9a01-4bfb-bf70-dff574ef1f0f","Type":"ContainerStarted","Data":"8629824dc012b337d159fcea59a15025ba550769df9004642f10d51cf52d92ec"} Nov 03 21:13:49.533929 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:49.533825 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e22abf-hncdr" podStartSLOduration=1.5338109869999998 podStartE2EDuration="1.533810987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:13:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:13:49.53129219 +0000 UTC m=+2064.756612005" watchObservedRunningTime="2026-11-03 21:13:49.533810987 +0000 UTC m=+2064.759130777" Nov 03 21:13:50.363778 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:50.363744 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-01c7a3-55bcd6997b-k8qhc"] Nov 03 21:13:50.423884 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:50.423849 4305 generic.go:356] "Generic (PLEG): container finished" podID="da6b7530-9a01-4bfb-bf70-dff574ef1f0f" containerID="d7f0175080aa64f5f922196e3f06e75d7c3d0cefc1992fc3fdc1859cf2744cf1" exitCode=0 Nov 03 21:13:50.424287 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:50.423948 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e22abf-hncdr" event={"ID":"da6b7530-9a01-4bfb-bf70-dff574ef1f0f","Type":"ContainerDied","Data":"d7f0175080aa64f5f922196e3f06e75d7c3d0cefc1992fc3fdc1859cf2744cf1"} Nov 03 21:13:50.475238 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:50.475205 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01c7a3-6479dcc9-xzqvp"] Nov 03 21:13:51.341527 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.341459 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc" Nov 03 21:13:51.361143 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.361115 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p" Nov 03 21:13:51.432895 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.432860 4305 generic.go:356] "Generic (PLEG): container finished" podID="e4b14774-55f2-4fad-88ad-74d08bf328f7" containerID="2abb835f949043e4a77b448f621626fa8dd5fca6fc7a6fd8ddc5da199fc88179" exitCode=0 Nov 03 21:13:51.433349 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.432912 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p" Nov 03 21:13:51.433349 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.432947 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p" event={"ID":"e4b14774-55f2-4fad-88ad-74d08bf328f7","Type":"ContainerDied","Data":"2abb835f949043e4a77b448f621626fa8dd5fca6fc7a6fd8ddc5da199fc88179"} Nov 03 21:13:51.433349 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.433006 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p" event={"ID":"e4b14774-55f2-4fad-88ad-74d08bf328f7","Type":"ContainerDied","Data":"7a015d42e81af71e1aa9f387216ec123075116947fc600dc5d968b14e1558ca4"} Nov 03 21:13:51.433349 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.433038 4305 scope.go:122] "RemoveContainer" containerID="2abb835f949043e4a77b448f621626fa8dd5fca6fc7a6fd8ddc5da199fc88179" Nov 03 21:13:51.434471 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.434411 4305 generic.go:356] "Generic (PLEG): container finished" podID="6a908050-5ca3-4977-b279-67965ac30de1" containerID="fdddee4a24fda9da49e1e23c26953af05b709276af091e6cf4da8eb05fe626b9" exitCode=0 Nov 03 21:13:51.434597 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.434466 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc" event={"ID":"6a908050-5ca3-4977-b279-67965ac30de1","Type":"ContainerDied","Data":"fdddee4a24fda9da49e1e23c26953af05b709276af091e6cf4da8eb05fe626b9"} Nov 03 21:13:51.434597 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.434495 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc" Nov 03 21:13:51.434597 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.434540 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc" event={"ID":"6a908050-5ca3-4977-b279-67965ac30de1","Type":"ContainerDied","Data":"5af04a7685814a46e482885deb0a3fd73afb8f8f9714114124f737fd42cf996f"} Nov 03 21:13:51.443959 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.443935 4305 scope.go:122] "RemoveContainer" containerID="b25dde8b6a1675398d99debfe7fc44da9de0edb7814bcfe8cb24683fafbc35d3" Nov 03 21:13:51.453293 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.453264 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4b14774-55f2-4fad-88ad-74d08bf328f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4b14774-55f2-4fad-88ad-74d08bf328f7-istio-envoy\") pod \"e4b14774-55f2-4fad-88ad-74d08bf328f7\" (UID: \"e4b14774-55f2-4fad-88ad-74d08bf328f7\") " Nov 03 21:13:51.453456 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.453358 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4b14774-55f2-4fad-88ad-74d08bf328f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b14774-55f2-4fad-88ad-74d08bf328f7-workload-socket\") pod \"e4b14774-55f2-4fad-88ad-74d08bf328f7\" (UID: \"e4b14774-55f2-4fad-88ad-74d08bf328f7\") " Nov 03 21:13:51.453567 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.453411 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4b14774-55f2-4fad-88ad-74d08bf328f7-kube-api-access-fkvpg\" (UniqueName: \"kubernetes.io/projected/e4b14774-55f2-4fad-88ad-74d08bf328f7-kube-api-access-fkvpg\") pod \"e4b14774-55f2-4fad-88ad-74d08bf328f7\" (UID: \"e4b14774-55f2-4fad-88ad-74d08bf328f7\") " Nov 03 21:13:51.453636 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.453625 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a908050-5ca3-4977-b279-67965ac30de1-istio-token\" (UniqueName: \"kubernetes.io/projected/6a908050-5ca3-4977-b279-67965ac30de1-istio-token\") pod \"6a908050-5ca3-4977-b279-67965ac30de1\" (UID: \"6a908050-5ca3-4977-b279-67965ac30de1\") " Nov 03 21:13:51.453694 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.453682 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a908050-5ca3-4977-b279-67965ac30de1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a908050-5ca3-4977-b279-67965ac30de1-istio-data\") pod \"6a908050-5ca3-4977-b279-67965ac30de1\" (UID: \"6a908050-5ca3-4977-b279-67965ac30de1\") " Nov 03 21:13:51.453758 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.453748 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6a908050-5ca3-4977-b279-67965ac30de1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a908050-5ca3-4977-b279-67965ac30de1-istio-podinfo\") pod \"6a908050-5ca3-4977-b279-67965ac30de1\" (UID: \"6a908050-5ca3-4977-b279-67965ac30de1\") " Nov 03 21:13:51.453818 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.453748 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4b14774-55f2-4fad-88ad-74d08bf328f7-workload-socket" pod "e4b14774-55f2-4fad-88ad-74d08bf328f7" (UID: "e4b14774-55f2-4fad-88ad-74d08bf328f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:13:51.453818 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.453782 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a908050-5ca3-4977-b279-67965ac30de1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a908050-5ca3-4977-b279-67965ac30de1-workload-socket\") pod \"6a908050-5ca3-4977-b279-67965ac30de1\" (UID: \"6a908050-5ca3-4977-b279-67965ac30de1\") " Nov 03 21:13:51.453978 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.453816 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a908050-5ca3-4977-b279-67965ac30de1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a908050-5ca3-4977-b279-67965ac30de1-credential-socket\") pod \"6a908050-5ca3-4977-b279-67965ac30de1\" (UID: \"6a908050-5ca3-4977-b279-67965ac30de1\") " Nov 03 21:13:51.453978 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.453853 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4b14774-55f2-4fad-88ad-74d08bf328f7-istio-token\" (UniqueName: \"kubernetes.io/projected/e4b14774-55f2-4fad-88ad-74d08bf328f7-istio-token\") pod \"e4b14774-55f2-4fad-88ad-74d08bf328f7\" (UID: \"e4b14774-55f2-4fad-88ad-74d08bf328f7\") " Nov 03 21:13:51.453978 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.453900 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e4b14774-55f2-4fad-88ad-74d08bf328f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4b14774-55f2-4fad-88ad-74d08bf328f7-istio-podinfo\") pod \"e4b14774-55f2-4fad-88ad-74d08bf328f7\" (UID: \"e4b14774-55f2-4fad-88ad-74d08bf328f7\") " Nov 03 21:13:51.453978 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.453940 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a908050-5ca3-4977-b279-67965ac30de1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a908050-5ca3-4977-b279-67965ac30de1-workload-certs\") pod \"6a908050-5ca3-4977-b279-67965ac30de1\" (UID: \"6a908050-5ca3-4977-b279-67965ac30de1\") " Nov 03 21:13:51.454224 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.454007 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a908050-5ca3-4977-b279-67965ac30de1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a908050-5ca3-4977-b279-67965ac30de1-istio-envoy\") pod \"6a908050-5ca3-4977-b279-67965ac30de1\" (UID: \"6a908050-5ca3-4977-b279-67965ac30de1\") " Nov 03 21:13:51.456738 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.454664 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4b14774-55f2-4fad-88ad-74d08bf328f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4b14774-55f2-4fad-88ad-74d08bf328f7-istio-data\") pod \"e4b14774-55f2-4fad-88ad-74d08bf328f7\" (UID: \"e4b14774-55f2-4fad-88ad-74d08bf328f7\") " Nov 03 21:13:51.456738 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.454802 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4b14774-55f2-4fad-88ad-74d08bf328f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4b14774-55f2-4fad-88ad-74d08bf328f7-workload-certs\") pod \"e4b14774-55f2-4fad-88ad-74d08bf328f7\" (UID: \"e4b14774-55f2-4fad-88ad-74d08bf328f7\") " Nov 03 21:13:51.456738 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.454830 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a908050-5ca3-4977-b279-67965ac30de1-istio-data" pod "6a908050-5ca3-4977-b279-67965ac30de1" (UID: "6a908050-5ca3-4977-b279-67965ac30de1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:13:51.456738 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.454861 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4b14774-55f2-4fad-88ad-74d08bf328f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b14774-55f2-4fad-88ad-74d08bf328f7-credential-socket\") pod \"e4b14774-55f2-4fad-88ad-74d08bf328f7\" (UID: \"e4b14774-55f2-4fad-88ad-74d08bf328f7\") " Nov 03 21:13:51.456738 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.454905 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a908050-5ca3-4977-b279-67965ac30de1-kube-api-access-2rgx2\" (UniqueName: \"kubernetes.io/projected/6a908050-5ca3-4977-b279-67965ac30de1-kube-api-access-2rgx2\") pod \"6a908050-5ca3-4977-b279-67965ac30de1\" (UID: \"6a908050-5ca3-4977-b279-67965ac30de1\") " Nov 03 21:13:51.456738 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.454963 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4b14774-55f2-4fad-88ad-74d08bf328f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4b14774-55f2-4fad-88ad-74d08bf328f7-istiod-ca-cert\") pod \"e4b14774-55f2-4fad-88ad-74d08bf328f7\" (UID: \"e4b14774-55f2-4fad-88ad-74d08bf328f7\") " Nov 03 21:13:51.456738 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.455002 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a908050-5ca3-4977-b279-67965ac30de1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a908050-5ca3-4977-b279-67965ac30de1-istiod-ca-cert\") pod \"6a908050-5ca3-4977-b279-67965ac30de1\" (UID: \"6a908050-5ca3-4977-b279-67965ac30de1\") " Nov 03 21:13:51.456738 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.455361 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4b14774-55f2-4fad-88ad-74d08bf328f7-workload-certs" pod "e4b14774-55f2-4fad-88ad-74d08bf328f7" (UID: "e4b14774-55f2-4fad-88ad-74d08bf328f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:13:51.456738 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.455505 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a908050-5ca3-4977-b279-67965ac30de1-credential-socket" pod "6a908050-5ca3-4977-b279-67965ac30de1" (UID: "6a908050-5ca3-4977-b279-67965ac30de1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:13:51.456738 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.455804 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a908050-5ca3-4977-b279-67965ac30de1-workload-certs" pod "6a908050-5ca3-4977-b279-67965ac30de1" (UID: "6a908050-5ca3-4977-b279-67965ac30de1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:13:51.456738 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.455806 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a908050-5ca3-4977-b279-67965ac30de1-istiod-ca-cert" pod "6a908050-5ca3-4977-b279-67965ac30de1" (UID: "6a908050-5ca3-4977-b279-67965ac30de1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:13:51.456738 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.455833 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4b14774-55f2-4fad-88ad-74d08bf328f7-workload-certs\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:51.456738 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.455860 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b14774-55f2-4fad-88ad-74d08bf328f7-workload-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:51.456738 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.455862 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4b14774-55f2-4fad-88ad-74d08bf328f7-istio-data" pod "e4b14774-55f2-4fad-88ad-74d08bf328f7" (UID: "e4b14774-55f2-4fad-88ad-74d08bf328f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:13:51.456738 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.455876 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a908050-5ca3-4977-b279-67965ac30de1-istio-data\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:51.456738 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.455890 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a908050-5ca3-4977-b279-67965ac30de1-credential-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:51.456738 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.455901 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a908050-5ca3-4977-b279-67965ac30de1-workload-socket" pod "6a908050-5ca3-4977-b279-67965ac30de1" (UID: "6a908050-5ca3-4977-b279-67965ac30de1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:13:51.456738 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.456103 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4b14774-55f2-4fad-88ad-74d08bf328f7-credential-socket" pod "e4b14774-55f2-4fad-88ad-74d08bf328f7" (UID: "e4b14774-55f2-4fad-88ad-74d08bf328f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:13:51.456738 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.456144 4305 scope.go:122] "RemoveContainer" containerID="2abb835f949043e4a77b448f621626fa8dd5fca6fc7a6fd8ddc5da199fc88179" Nov 03 21:13:51.456738 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.456175 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4b14774-55f2-4fad-88ad-74d08bf328f7-istiod-ca-cert" pod "e4b14774-55f2-4fad-88ad-74d08bf328f7" (UID: "e4b14774-55f2-4fad-88ad-74d08bf328f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:13:51.456738 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:13:51.456538 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2abb835f949043e4a77b448f621626fa8dd5fca6fc7a6fd8ddc5da199fc88179\": container with ID starting with 2abb835f949043e4a77b448f621626fa8dd5fca6fc7a6fd8ddc5da199fc88179 not found: ID does not exist" containerID="2abb835f949043e4a77b448f621626fa8dd5fca6fc7a6fd8ddc5da199fc88179" Nov 03 21:13:51.456738 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.456573 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2abb835f949043e4a77b448f621626fa8dd5fca6fc7a6fd8ddc5da199fc88179"} err="failed to get container status \"2abb835f949043e4a77b448f621626fa8dd5fca6fc7a6fd8ddc5da199fc88179\": rpc error: code = NotFound desc = could not find container \"2abb835f949043e4a77b448f621626fa8dd5fca6fc7a6fd8ddc5da199fc88179\": container with ID starting with 2abb835f949043e4a77b448f621626fa8dd5fca6fc7a6fd8ddc5da199fc88179 not found: ID does not exist" Nov 03 21:13:51.456738 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.456600 4305 scope.go:122] "RemoveContainer" containerID="b25dde8b6a1675398d99debfe7fc44da9de0edb7814bcfe8cb24683fafbc35d3" Nov 03 21:13:51.458796 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:13:51.456928 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b25dde8b6a1675398d99debfe7fc44da9de0edb7814bcfe8cb24683fafbc35d3\": container with ID starting with b25dde8b6a1675398d99debfe7fc44da9de0edb7814bcfe8cb24683fafbc35d3 not found: ID does not exist" containerID="b25dde8b6a1675398d99debfe7fc44da9de0edb7814bcfe8cb24683fafbc35d3" Nov 03 21:13:51.458796 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.456947 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a908050-5ca3-4977-b279-67965ac30de1-istio-token" pod "6a908050-5ca3-4977-b279-67965ac30de1" (UID: "6a908050-5ca3-4977-b279-67965ac30de1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:13:51.458796 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.456962 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b25dde8b6a1675398d99debfe7fc44da9de0edb7814bcfe8cb24683fafbc35d3"} err="failed to get container status \"b25dde8b6a1675398d99debfe7fc44da9de0edb7814bcfe8cb24683fafbc35d3\": rpc error: code = NotFound desc = could not find container \"b25dde8b6a1675398d99debfe7fc44da9de0edb7814bcfe8cb24683fafbc35d3\": container with ID starting with b25dde8b6a1675398d99debfe7fc44da9de0edb7814bcfe8cb24683fafbc35d3 not found: ID does not exist" Nov 03 21:13:51.458796 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.456986 4305 scope.go:122] "RemoveContainer" containerID="fdddee4a24fda9da49e1e23c26953af05b709276af091e6cf4da8eb05fe626b9" Nov 03 21:13:51.458796 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.457419 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4b14774-55f2-4fad-88ad-74d08bf328f7-istio-envoy" pod "e4b14774-55f2-4fad-88ad-74d08bf328f7" (UID: "e4b14774-55f2-4fad-88ad-74d08bf328f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:13:51.458796 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.457731 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a908050-5ca3-4977-b279-67965ac30de1-istio-podinfo" pod "6a908050-5ca3-4977-b279-67965ac30de1" (UID: "6a908050-5ca3-4977-b279-67965ac30de1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:13:51.458796 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.458254 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a908050-5ca3-4977-b279-67965ac30de1-istio-envoy" pod "6a908050-5ca3-4977-b279-67965ac30de1" (UID: "6a908050-5ca3-4977-b279-67965ac30de1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:13:51.458796 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.458401 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e4b14774-55f2-4fad-88ad-74d08bf328f7-istio-podinfo" pod "e4b14774-55f2-4fad-88ad-74d08bf328f7" (UID: "e4b14774-55f2-4fad-88ad-74d08bf328f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:13:51.458796 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.458422 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4b14774-55f2-4fad-88ad-74d08bf328f7-kube-api-access-fkvpg" pod "e4b14774-55f2-4fad-88ad-74d08bf328f7" (UID: "e4b14774-55f2-4fad-88ad-74d08bf328f7"). InnerVolumeSpecName "kube-api-access-fkvpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:13:51.459925 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.459887 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a908050-5ca3-4977-b279-67965ac30de1-kube-api-access-2rgx2" pod "6a908050-5ca3-4977-b279-67965ac30de1" (UID: "6a908050-5ca3-4977-b279-67965ac30de1"). InnerVolumeSpecName "kube-api-access-2rgx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:13:51.460265 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.460242 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4b14774-55f2-4fad-88ad-74d08bf328f7-istio-token" pod "e4b14774-55f2-4fad-88ad-74d08bf328f7" (UID: "e4b14774-55f2-4fad-88ad-74d08bf328f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:13:51.472350 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.472315 4305 scope.go:122] "RemoveContainer" containerID="fdddee4a24fda9da49e1e23c26953af05b709276af091e6cf4da8eb05fe626b9" Nov 03 21:13:51.472866 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:13:51.472796 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdddee4a24fda9da49e1e23c26953af05b709276af091e6cf4da8eb05fe626b9\": container with ID starting with fdddee4a24fda9da49e1e23c26953af05b709276af091e6cf4da8eb05fe626b9 not found: ID does not exist" containerID="fdddee4a24fda9da49e1e23c26953af05b709276af091e6cf4da8eb05fe626b9" Nov 03 21:13:51.472866 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.472840 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdddee4a24fda9da49e1e23c26953af05b709276af091e6cf4da8eb05fe626b9"} err="failed to get container status \"fdddee4a24fda9da49e1e23c26953af05b709276af091e6cf4da8eb05fe626b9\": rpc error: code = NotFound desc = could not find container \"fdddee4a24fda9da49e1e23c26953af05b709276af091e6cf4da8eb05fe626b9\": container with ID starting with fdddee4a24fda9da49e1e23c26953af05b709276af091e6cf4da8eb05fe626b9 not found: ID does not exist" Nov 03 21:13:51.557286 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.557230 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a908050-5ca3-4977-b279-67965ac30de1-istio-podinfo\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:51.557286 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.557289 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a908050-5ca3-4977-b279-67965ac30de1-workload-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:51.557762 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.557308 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4b14774-55f2-4fad-88ad-74d08bf328f7-istio-token\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:51.557762 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.557324 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4b14774-55f2-4fad-88ad-74d08bf328f7-istio-podinfo\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:51.557762 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.557354 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a908050-5ca3-4977-b279-67965ac30de1-workload-certs\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:51.557762 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.557367 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a908050-5ca3-4977-b279-67965ac30de1-istio-envoy\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:51.557762 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.557375 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4b14774-55f2-4fad-88ad-74d08bf328f7-istio-data\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:51.557762 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.557383 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b14774-55f2-4fad-88ad-74d08bf328f7-credential-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:51.557762 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.557393 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rgx2\" (UniqueName: \"kubernetes.io/projected/6a908050-5ca3-4977-b279-67965ac30de1-kube-api-access-2rgx2\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:51.557762 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.557402 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4b14774-55f2-4fad-88ad-74d08bf328f7-istiod-ca-cert\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:51.557762 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.557412 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a908050-5ca3-4977-b279-67965ac30de1-istiod-ca-cert\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:51.557762 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.557420 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4b14774-55f2-4fad-88ad-74d08bf328f7-istio-envoy\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:51.557762 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.557464 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkvpg\" (UniqueName: \"kubernetes.io/projected/e4b14774-55f2-4fad-88ad-74d08bf328f7-kube-api-access-fkvpg\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:51.557762 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.557478 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a908050-5ca3-4977-b279-67965ac30de1-istio-token\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:51.614778 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.614752 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e22abf-hncdr" Nov 03 21:13:51.657995 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.657960 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da6b7530-9a01-4bfb-bf70-dff574ef1f0f-kube-api-access-zwd9w\" (UniqueName: \"kubernetes.io/projected/da6b7530-9a01-4bfb-bf70-dff574ef1f0f-kube-api-access-zwd9w\") pod \"da6b7530-9a01-4bfb-bf70-dff574ef1f0f\" (UID: \"da6b7530-9a01-4bfb-bf70-dff574ef1f0f\") " Nov 03 21:13:51.658199 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.658134 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da6b7530-9a01-4bfb-bf70-dff574ef1f0f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da6b7530-9a01-4bfb-bf70-dff574ef1f0f-maas-api-xks-ca\") pod \"da6b7530-9a01-4bfb-bf70-dff574ef1f0f\" (UID: \"da6b7530-9a01-4bfb-bf70-dff574ef1f0f\") " Nov 03 21:13:51.658199 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.658174 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da6b7530-9a01-4bfb-bf70-dff574ef1f0f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da6b7530-9a01-4bfb-bf70-dff574ef1f0f-maas-api-openshift-ca\") pod \"da6b7530-9a01-4bfb-bf70-dff574ef1f0f\" (UID: \"da6b7530-9a01-4bfb-bf70-dff574ef1f0f\") " Nov 03 21:13:51.658594 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.658568 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da6b7530-9a01-4bfb-bf70-dff574ef1f0f-maas-api-openshift-ca" pod "da6b7530-9a01-4bfb-bf70-dff574ef1f0f" (UID: "da6b7530-9a01-4bfb-bf70-dff574ef1f0f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:13:51.660280 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.660259 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da6b7530-9a01-4bfb-bf70-dff574ef1f0f-kube-api-access-zwd9w" pod "da6b7530-9a01-4bfb-bf70-dff574ef1f0f" (UID: "da6b7530-9a01-4bfb-bf70-dff574ef1f0f"). InnerVolumeSpecName "kube-api-access-zwd9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:13:51.660341 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.660314 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da6b7530-9a01-4bfb-bf70-dff574ef1f0f-maas-api-xks-ca" pod "da6b7530-9a01-4bfb-bf70-dff574ef1f0f" (UID: "da6b7530-9a01-4bfb-bf70-dff574ef1f0f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:13:51.768458 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.763736 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwd9w\" (UniqueName: \"kubernetes.io/projected/da6b7530-9a01-4bfb-bf70-dff574ef1f0f-kube-api-access-zwd9w\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:51.768458 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.763780 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da6b7530-9a01-4bfb-bf70-dff574ef1f0f-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:51.768458 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.763797 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da6b7530-9a01-4bfb-bf70-dff574ef1f0f-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:51.886540 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.884529 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p"] Nov 03 21:13:51.901637 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.900799 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-ttq8p"] Nov 03 21:13:51.986028 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.985989 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc"] Nov 03 21:13:51.991727 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:51.991694 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-d01cef-openshift-default-6d9778d4b5-8rjvc"] Nov 03 21:13:52.450096 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:52.450056 4305 generic.go:356] "Generic (PLEG): container finished" podID="953874ed-6376-4775-b514-b09ecb325824" containerID="3e9b081e86ee58626087431ea3a50f9d3b1d0fdd64224897f303ba82806b26ff" exitCode=0 Nov 03 21:13:52.450096 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:52.450090 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr" event={"ID":"953874ed-6376-4775-b514-b09ecb325824","Type":"ContainerDied","Data":"3e9b081e86ee58626087431ea3a50f9d3b1d0fdd64224897f303ba82806b26ff"} Nov 03 21:13:52.450726 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:52.450157 4305 scope.go:122] "RemoveContainer" containerID="ce43d975478cc8a99b640aee6c05a90a6cf327d4f1d6852ae96596c44910f64d" Nov 03 21:13:52.452858 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:52.452831 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e22abf-hncdr" event={"ID":"da6b7530-9a01-4bfb-bf70-dff574ef1f0f","Type":"ContainerDied","Data":"8629824dc012b337d159fcea59a15025ba550769df9004642f10d51cf52d92ec"} Nov 03 21:13:52.452984 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:52.452859 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e22abf-hncdr" Nov 03 21:13:52.452984 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:52.452878 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8629824dc012b337d159fcea59a15025ba550769df9004642f10d51cf52d92ec" Nov 03 21:13:52.538829 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:52.538791 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr" Nov 03 21:13:52.590352 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:52.590321 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/953874ed-6376-4775-b514-b09ecb325824-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/953874ed-6376-4775-b514-b09ecb325824-workload-socket\") pod \"953874ed-6376-4775-b514-b09ecb325824\" (UID: \"953874ed-6376-4775-b514-b09ecb325824\") " Nov 03 21:13:52.590558 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:52.590393 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/953874ed-6376-4775-b514-b09ecb325824-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/953874ed-6376-4775-b514-b09ecb325824-istio-podinfo\") pod \"953874ed-6376-4775-b514-b09ecb325824\" (UID: \"953874ed-6376-4775-b514-b09ecb325824\") " Nov 03 21:13:52.590558 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:52.590487 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/953874ed-6376-4775-b514-b09ecb325824-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/953874ed-6376-4775-b514-b09ecb325824-workload-certs\") pod \"953874ed-6376-4775-b514-b09ecb325824\" (UID: \"953874ed-6376-4775-b514-b09ecb325824\") " Nov 03 21:13:52.590693 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:52.590566 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/953874ed-6376-4775-b514-b09ecb325824-kube-api-access-sjqvc\" (UniqueName: \"kubernetes.io/projected/953874ed-6376-4775-b514-b09ecb325824-kube-api-access-sjqvc\") pod \"953874ed-6376-4775-b514-b09ecb325824\" (UID: \"953874ed-6376-4775-b514-b09ecb325824\") " Nov 03 21:13:52.590693 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:52.590588 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/953874ed-6376-4775-b514-b09ecb325824-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/953874ed-6376-4775-b514-b09ecb325824-istio-envoy\") pod \"953874ed-6376-4775-b514-b09ecb325824\" (UID: \"953874ed-6376-4775-b514-b09ecb325824\") " Nov 03 21:13:52.590693 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:52.590617 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/953874ed-6376-4775-b514-b09ecb325824-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/953874ed-6376-4775-b514-b09ecb325824-credential-socket\") pod \"953874ed-6376-4775-b514-b09ecb325824\" (UID: \"953874ed-6376-4775-b514-b09ecb325824\") " Nov 03 21:13:52.590693 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:52.590638 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/953874ed-6376-4775-b514-b09ecb325824-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/953874ed-6376-4775-b514-b09ecb325824-istiod-ca-cert\") pod \"953874ed-6376-4775-b514-b09ecb325824\" (UID: \"953874ed-6376-4775-b514-b09ecb325824\") " Nov 03 21:13:52.590693 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:52.590650 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/953874ed-6376-4775-b514-b09ecb325824-workload-socket" pod "953874ed-6376-4775-b514-b09ecb325824" (UID: "953874ed-6376-4775-b514-b09ecb325824"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:13:52.590944 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:52.590754 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/953874ed-6376-4775-b514-b09ecb325824-istio-token\" (UniqueName: \"kubernetes.io/projected/953874ed-6376-4775-b514-b09ecb325824-istio-token\") pod \"953874ed-6376-4775-b514-b09ecb325824\" (UID: \"953874ed-6376-4775-b514-b09ecb325824\") " Nov 03 21:13:52.590944 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:52.590904 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/953874ed-6376-4775-b514-b09ecb325824-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/953874ed-6376-4775-b514-b09ecb325824-istio-data\") pod \"953874ed-6376-4775-b514-b09ecb325824\" (UID: \"953874ed-6376-4775-b514-b09ecb325824\") " Nov 03 21:13:52.591064 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:52.591041 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/953874ed-6376-4775-b514-b09ecb325824-istiod-ca-cert" pod "953874ed-6376-4775-b514-b09ecb325824" (UID: "953874ed-6376-4775-b514-b09ecb325824"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:13:52.591119 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:52.591048 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/953874ed-6376-4775-b514-b09ecb325824-workload-certs" pod "953874ed-6376-4775-b514-b09ecb325824" (UID: "953874ed-6376-4775-b514-b09ecb325824"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:13:52.591174 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:52.591150 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/953874ed-6376-4775-b514-b09ecb325824-credential-socket" pod "953874ed-6376-4775-b514-b09ecb325824" (UID: "953874ed-6376-4775-b514-b09ecb325824"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:13:52.591562 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:52.591533 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/953874ed-6376-4775-b514-b09ecb325824-istio-data" pod "953874ed-6376-4775-b514-b09ecb325824" (UID: "953874ed-6376-4775-b514-b09ecb325824"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:13:52.591721 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:52.591659 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/953874ed-6376-4775-b514-b09ecb325824-credential-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:52.591721 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:52.591680 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/953874ed-6376-4775-b514-b09ecb325824-istiod-ca-cert\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:52.591721 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:52.591695 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/953874ed-6376-4775-b514-b09ecb325824-istio-data\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:52.591721 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:52.591710 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/953874ed-6376-4775-b514-b09ecb325824-workload-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:52.591895 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:52.591724 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/953874ed-6376-4775-b514-b09ecb325824-workload-certs\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:52.594683 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:52.594658 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/953874ed-6376-4775-b514-b09ecb325824-istio-envoy" pod "953874ed-6376-4775-b514-b09ecb325824" (UID: "953874ed-6376-4775-b514-b09ecb325824"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:13:52.594683 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:52.594667 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/953874ed-6376-4775-b514-b09ecb325824-istio-podinfo" pod "953874ed-6376-4775-b514-b09ecb325824" (UID: "953874ed-6376-4775-b514-b09ecb325824"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:13:52.594868 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:52.594840 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/953874ed-6376-4775-b514-b09ecb325824-istio-token" pod "953874ed-6376-4775-b514-b09ecb325824" (UID: "953874ed-6376-4775-b514-b09ecb325824"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:13:52.594970 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:52.594875 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/953874ed-6376-4775-b514-b09ecb325824-kube-api-access-sjqvc" pod "953874ed-6376-4775-b514-b09ecb325824" (UID: "953874ed-6376-4775-b514-b09ecb325824"). InnerVolumeSpecName "kube-api-access-sjqvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:13:52.692866 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:52.692825 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjqvc\" (UniqueName: \"kubernetes.io/projected/953874ed-6376-4775-b514-b09ecb325824-kube-api-access-sjqvc\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:52.692866 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:52.692867 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/953874ed-6376-4775-b514-b09ecb325824-istio-envoy\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:52.693103 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:52.692882 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/953874ed-6376-4775-b514-b09ecb325824-istio-token\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:52.693103 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:52.692896 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/953874ed-6376-4775-b514-b09ecb325824-istio-podinfo\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:52.719057 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:52.719021 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8"] Nov 03 21:13:52.752460 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:52.752406 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq"] Nov 03 21:13:52.895590 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:52.895561 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a908050-5ca3-4977-b279-67965ac30de1" path="/var/lib/kubelet/pods/6a908050-5ca3-4977-b279-67965ac30de1/volumes" Nov 03 21:13:52.896028 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:52.896013 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4b14774-55f2-4fad-88ad-74d08bf328f7" path="/var/lib/kubelet/pods/e4b14774-55f2-4fad-88ad-74d08bf328f7/volumes" Nov 03 21:13:53.463418 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:53.463377 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr" Nov 03 21:13:53.464077 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:53.463381 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr" event={"ID":"953874ed-6376-4775-b514-b09ecb325824","Type":"ContainerDied","Data":"77b81da472f95c78595e484469689e647bf1d955fc3c4f7c2729b0c1b62085bc"} Nov 03 21:13:53.464077 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:53.463537 4305 scope.go:122] "RemoveContainer" containerID="3e9b081e86ee58626087431ea3a50f9d3b1d0fdd64224897f303ba82806b26ff" Nov 03 21:13:53.631251 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:53.631216 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr"] Nov 03 21:13:53.680885 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:53.680851 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-32c41e-openshift-default-56bd575b55-hzwdr"] Nov 03 21:13:53.860490 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:53.860390 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs"] Nov 03 21:13:53.860668 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:53.860644 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs" podUID="e4063115-7846-42ea-8bea-4684c27b132a" containerName="istio-proxy" containerID="cri-o://9795139d2fdc6878b2ca6a18c3b3908b403cd54e700ce49c53546f48b6ace684" gracePeriod=30 Nov 03 21:13:54.896794 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:54.896759 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="953874ed-6376-4775-b514-b09ecb325824" path="/var/lib/kubelet/pods/953874ed-6376-4775-b514-b09ecb325824/volumes" Nov 03 21:13:54.933896 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:54.933863 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-12a423-2hvxd"] Nov 03 21:13:54.935423 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:54.935394 4305 cpu_manager.go:414] "containerMap: removing container" podUID="e4b14774-55f2-4fad-88ad-74d08bf328f7" containerName="istio-proxy" Nov 03 21:13:54.935622 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:54.935606 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4b14774-55f2-4fad-88ad-74d08bf328f7" containerName="istio-proxy" Nov 03 21:13:54.935753 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:54.935741 4305 cpu_manager.go:414] "containerMap: removing container" podUID="953874ed-6376-4775-b514-b09ecb325824" containerName="istio-proxy" Nov 03 21:13:54.935850 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:54.935827 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="953874ed-6376-4775-b514-b09ecb325824" containerName="istio-proxy" Nov 03 21:13:54.936000 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:54.935865 4305 cpu_manager.go:414] "containerMap: removing container" podUID="953874ed-6376-4775-b514-b09ecb325824" containerName="istio-proxy" Nov 03 21:13:54.936000 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:54.935875 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="953874ed-6376-4775-b514-b09ecb325824" containerName="istio-proxy" Nov 03 21:13:54.936000 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:54.935897 4305 cpu_manager.go:414] "containerMap: removing container" podUID="e4b14774-55f2-4fad-88ad-74d08bf328f7" containerName="istio-proxy" Nov 03 21:13:54.936000 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:54.935905 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4b14774-55f2-4fad-88ad-74d08bf328f7" containerName="istio-proxy" Nov 03 21:13:54.936000 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:54.935942 4305 cpu_manager.go:414] "containerMap: removing container" podUID="da6b7530-9a01-4bfb-bf70-dff574ef1f0f" containerName="revoke-keys" Nov 03 21:13:54.936000 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:54.935951 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da6b7530-9a01-4bfb-bf70-dff574ef1f0f" containerName="revoke-keys" Nov 03 21:13:54.936000 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:54.935976 4305 cpu_manager.go:414] "containerMap: removing container" podUID="6a908050-5ca3-4977-b279-67965ac30de1" containerName="istio-proxy" Nov 03 21:13:54.936000 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:54.935983 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a908050-5ca3-4977-b279-67965ac30de1" containerName="istio-proxy" Nov 03 21:13:54.936327 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:54.936154 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="da6b7530-9a01-4bfb-bf70-dff574ef1f0f" containerName="revoke-keys" Nov 03 21:13:54.936327 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:54.936171 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a908050-5ca3-4977-b279-67965ac30de1" containerName="istio-proxy" Nov 03 21:13:54.936327 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:54.936189 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4b14774-55f2-4fad-88ad-74d08bf328f7" containerName="istio-proxy" Nov 03 21:13:54.936327 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:54.936199 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="953874ed-6376-4775-b514-b09ecb325824" containerName="istio-proxy" Nov 03 21:13:54.936327 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:54.936211 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="953874ed-6376-4775-b514-b09ecb325824" containerName="istio-proxy" Nov 03 21:13:54.936327 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:54.936221 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4b14774-55f2-4fad-88ad-74d08bf328f7" containerName="istio-proxy" Nov 03 21:13:54.941392 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:54.941367 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-12a423-2hvxd" Nov 03 21:13:54.993581 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:54.993550 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:13:54.993817 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:54.993798 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:13:55.030702 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:55.030664 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q7qs\" (UniqueName: \"kubernetes.io/projected/9b488427-d78e-41c3-848d-b5cf765a1e83-kube-api-access-2q7qs\") pod \"maas-api-revoke-keys-e2e-worker-w6-12a423-2hvxd\" (UID: \"9b488427-d78e-41c3-848d-b5cf765a1e83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-12a423-2hvxd" Nov 03 21:13:55.030869 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:55.030809 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b488427-d78e-41c3-848d-b5cf765a1e83-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-12a423-2hvxd\" (UID: \"9b488427-d78e-41c3-848d-b5cf765a1e83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-12a423-2hvxd" Nov 03 21:13:55.030932 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:55.030901 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b488427-d78e-41c3-848d-b5cf765a1e83-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-12a423-2hvxd\" (UID: \"9b488427-d78e-41c3-848d-b5cf765a1e83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-12a423-2hvxd" Nov 03 21:13:55.043186 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:55.043052 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:13:55.043186 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:55.043171 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77vtd\"" Nov 03 21:13:55.047409 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:55.047376 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-12a423-2hvxd"] Nov 03 21:13:55.132187 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:55.132143 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b488427-d78e-41c3-848d-b5cf765a1e83-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-12a423-2hvxd\" (UID: \"9b488427-d78e-41c3-848d-b5cf765a1e83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-12a423-2hvxd" Nov 03 21:13:55.132407 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:55.132278 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q7qs\" (UniqueName: \"kubernetes.io/projected/9b488427-d78e-41c3-848d-b5cf765a1e83-kube-api-access-2q7qs\") pod \"maas-api-revoke-keys-e2e-worker-w6-12a423-2hvxd\" (UID: \"9b488427-d78e-41c3-848d-b5cf765a1e83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-12a423-2hvxd" Nov 03 21:13:55.132407 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:55.132396 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b488427-d78e-41c3-848d-b5cf765a1e83-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-12a423-2hvxd\" (UID: \"9b488427-d78e-41c3-848d-b5cf765a1e83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-12a423-2hvxd" Nov 03 21:13:55.132854 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:55.132830 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b488427-d78e-41c3-848d-b5cf765a1e83-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-12a423-2hvxd\" (UID: \"9b488427-d78e-41c3-848d-b5cf765a1e83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-12a423-2hvxd" Nov 03 21:13:55.134917 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:55.134895 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b488427-d78e-41c3-848d-b5cf765a1e83-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-12a423-2hvxd\" (UID: \"9b488427-d78e-41c3-848d-b5cf765a1e83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-12a423-2hvxd" Nov 03 21:13:55.193322 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:55.193248 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q7qs\" (UniqueName: \"kubernetes.io/projected/9b488427-d78e-41c3-848d-b5cf765a1e83-kube-api-access-2q7qs\") pod \"maas-api-revoke-keys-e2e-worker-w6-12a423-2hvxd\" (UID: \"9b488427-d78e-41c3-848d-b5cf765a1e83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-12a423-2hvxd" Nov 03 21:13:55.256755 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:55.256712 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-12a423-2hvxd" Nov 03 21:13:55.364028 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:55.363990 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-01c7a3-55bcd6997b-k8qhc" podUID="7fb81be4-4690-45e1-947f-2de9f107d4d7" containerName="payload-processing" containerID="cri-o://39f3dd56347f4b50bdc9ddf83440d99d31ff1da3aab741b59af6b8fb62e7cb87" gracePeriod=25 Nov 03 21:13:55.475961 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:55.475862 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01c7a3-6479dcc9-xzqvp" podUID="eff346af-88ef-494d-8505-c8546c5c6f4e" containerName="payload-pre-processing" containerID="cri-o://2f6806542fb8a8a58fe50629a9b9615be05bc76c89cd6ce0da633a5ff581aed9" gracePeriod=25 Nov 03 21:13:55.481491 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:55.481467 4305 generic.go:356] "Generic (PLEG): container finished" podID="7fb81be4-4690-45e1-947f-2de9f107d4d7" containerID="39f3dd56347f4b50bdc9ddf83440d99d31ff1da3aab741b59af6b8fb62e7cb87" exitCode=0 Nov 03 21:13:55.481491 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:55.481478 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-01c7a3-55bcd6997b-k8qhc" event={"ID":"7fb81be4-4690-45e1-947f-2de9f107d4d7","Type":"ContainerDied","Data":"39f3dd56347f4b50bdc9ddf83440d99d31ff1da3aab741b59af6b8fb62e7cb87"} Nov 03 21:13:55.627978 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:55.627951 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-01c7a3-55bcd6997b-k8qhc" Nov 03 21:13:55.641417 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:55.641390 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-12a423-2hvxd"] Nov 03 21:13:55.642798 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:13:55.642771 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b488427_d78e_41c3_848d_b5cf765a1e83.slice/crio-56bea4722d022ca53edbffd328e53cae91fab35c927c7f75d2bc0bc0f7149c52 WatchSource:0}: Error finding container 56bea4722d022ca53edbffd328e53cae91fab35c927c7f75d2bc0bc0f7149c52: Status 404 returned error can't find the container with id 56bea4722d022ca53edbffd328e53cae91fab35c927c7f75d2bc0bc0f7149c52 Nov 03 21:13:55.740421 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:55.740336 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fb81be4-4690-45e1-947f-2de9f107d4d7-kube-api-access-n4t6b\" (UniqueName: \"kubernetes.io/projected/7fb81be4-4690-45e1-947f-2de9f107d4d7-kube-api-access-n4t6b\") pod \"7fb81be4-4690-45e1-947f-2de9f107d4d7\" (UID: \"7fb81be4-4690-45e1-947f-2de9f107d4d7\") " Nov 03 21:13:55.740608 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:55.740500 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fb81be4-4690-45e1-947f-2de9f107d4d7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fb81be4-4690-45e1-947f-2de9f107d4d7-plugins-config-volume\") pod \"7fb81be4-4690-45e1-947f-2de9f107d4d7\" (UID: \"7fb81be4-4690-45e1-947f-2de9f107d4d7\") " Nov 03 21:13:55.741017 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:55.740982 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fb81be4-4690-45e1-947f-2de9f107d4d7-plugins-config-volume" pod "7fb81be4-4690-45e1-947f-2de9f107d4d7" (UID: "7fb81be4-4690-45e1-947f-2de9f107d4d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:13:55.741119 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:55.741092 4305 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fb81be4-4690-45e1-947f-2de9f107d4d7-plugins-config-volume\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:55.742868 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:55.742822 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fb81be4-4690-45e1-947f-2de9f107d4d7-kube-api-access-n4t6b" pod "7fb81be4-4690-45e1-947f-2de9f107d4d7" (UID: "7fb81be4-4690-45e1-947f-2de9f107d4d7"). InnerVolumeSpecName "kube-api-access-n4t6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:13:55.758574 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:55.758547 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01c7a3-6479dcc9-xzqvp" Nov 03 21:13:55.841988 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:55.841942 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eff346af-88ef-494d-8505-c8546c5c6f4e-kube-api-access-t64zd\" (UniqueName: \"kubernetes.io/projected/eff346af-88ef-494d-8505-c8546c5c6f4e-kube-api-access-t64zd\") pod \"eff346af-88ef-494d-8505-c8546c5c6f4e\" (UID: \"eff346af-88ef-494d-8505-c8546c5c6f4e\") " Nov 03 21:13:55.842189 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:55.842030 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eff346af-88ef-494d-8505-c8546c5c6f4e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eff346af-88ef-494d-8505-c8546c5c6f4e-plugins-config-volume\") pod \"eff346af-88ef-494d-8505-c8546c5c6f4e\" (UID: \"eff346af-88ef-494d-8505-c8546c5c6f4e\") " Nov 03 21:13:55.842387 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:55.842360 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eff346af-88ef-494d-8505-c8546c5c6f4e-plugins-config-volume" pod "eff346af-88ef-494d-8505-c8546c5c6f4e" (UID: "eff346af-88ef-494d-8505-c8546c5c6f4e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:13:55.842387 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:55.842382 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4t6b\" (UniqueName: \"kubernetes.io/projected/7fb81be4-4690-45e1-947f-2de9f107d4d7-kube-api-access-n4t6b\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:55.852280 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:55.852248 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eff346af-88ef-494d-8505-c8546c5c6f4e-kube-api-access-t64zd" pod "eff346af-88ef-494d-8505-c8546c5c6f4e" (UID: "eff346af-88ef-494d-8505-c8546c5c6f4e"). InnerVolumeSpecName "kube-api-access-t64zd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:13:55.944149 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:55.944111 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t64zd\" (UniqueName: \"kubernetes.io/projected/eff346af-88ef-494d-8505-c8546c5c6f4e-kube-api-access-t64zd\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:55.944149 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:55.944143 4305 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eff346af-88ef-494d-8505-c8546c5c6f4e-plugins-config-volume\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:56.489962 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:56.489929 4305 generic.go:356] "Generic (PLEG): container finished" podID="eff346af-88ef-494d-8505-c8546c5c6f4e" containerID="2f6806542fb8a8a58fe50629a9b9615be05bc76c89cd6ce0da633a5ff581aed9" exitCode=0 Nov 03 21:13:56.490157 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:56.489980 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01c7a3-6479dcc9-xzqvp" Nov 03 21:13:56.490157 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:56.490009 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01c7a3-6479dcc9-xzqvp" event={"ID":"eff346af-88ef-494d-8505-c8546c5c6f4e","Type":"ContainerDied","Data":"2f6806542fb8a8a58fe50629a9b9615be05bc76c89cd6ce0da633a5ff581aed9"} Nov 03 21:13:56.490157 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:56.490068 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01c7a3-6479dcc9-xzqvp" event={"ID":"eff346af-88ef-494d-8505-c8546c5c6f4e","Type":"ContainerDied","Data":"042dc699f618e329bd7c89aef947bcc7173a1880399ec5c96c1a79b6c82da3fb"} Nov 03 21:13:56.490157 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:56.490096 4305 scope.go:122] "RemoveContainer" containerID="2f6806542fb8a8a58fe50629a9b9615be05bc76c89cd6ce0da633a5ff581aed9" Nov 03 21:13:56.492102 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:56.492072 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-12a423-2hvxd" event={"ID":"9b488427-d78e-41c3-848d-b5cf765a1e83","Type":"ContainerStarted","Data":"bb2a68447ca969d2a8aa6fd28ff9cd774d3af40c41d10c351a099fb5d20fe7bc"} Nov 03 21:13:56.492242 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:56.492126 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-12a423-2hvxd" event={"ID":"9b488427-d78e-41c3-848d-b5cf765a1e83","Type":"ContainerStarted","Data":"56bea4722d022ca53edbffd328e53cae91fab35c927c7f75d2bc0bc0f7149c52"} Nov 03 21:13:56.493731 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:56.493713 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-01c7a3-55bcd6997b-k8qhc" Nov 03 21:13:56.493848 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:56.493756 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-01c7a3-55bcd6997b-k8qhc" event={"ID":"7fb81be4-4690-45e1-947f-2de9f107d4d7","Type":"ContainerDied","Data":"91e7a462b8c819964c9739c2128f269f8c3f78efba174227ed73e47ef5cb1ef7"} Nov 03 21:13:56.502093 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:56.502071 4305 scope.go:122] "RemoveContainer" containerID="2f6806542fb8a8a58fe50629a9b9615be05bc76c89cd6ce0da633a5ff581aed9" Nov 03 21:13:56.502407 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:13:56.502388 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f6806542fb8a8a58fe50629a9b9615be05bc76c89cd6ce0da633a5ff581aed9\": container with ID starting with 2f6806542fb8a8a58fe50629a9b9615be05bc76c89cd6ce0da633a5ff581aed9 not found: ID does not exist" containerID="2f6806542fb8a8a58fe50629a9b9615be05bc76c89cd6ce0da633a5ff581aed9" Nov 03 21:13:56.502516 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:56.502414 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f6806542fb8a8a58fe50629a9b9615be05bc76c89cd6ce0da633a5ff581aed9"} err="failed to get container status \"2f6806542fb8a8a58fe50629a9b9615be05bc76c89cd6ce0da633a5ff581aed9\": rpc error: code = NotFound desc = could not find container \"2f6806542fb8a8a58fe50629a9b9615be05bc76c89cd6ce0da633a5ff581aed9\": container with ID starting with 2f6806542fb8a8a58fe50629a9b9615be05bc76c89cd6ce0da633a5ff581aed9 not found: ID does not exist" Nov 03 21:13:56.502516 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:56.502448 4305 scope.go:122] "RemoveContainer" containerID="39f3dd56347f4b50bdc9ddf83440d99d31ff1da3aab741b59af6b8fb62e7cb87" Nov 03 21:13:56.679845 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:56.679806 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01c7a3-6479dcc9-xzqvp"] Nov 03 21:13:56.685662 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:56.685609 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-01c7a3-6479dcc9-xzqvp"] Nov 03 21:13:56.797189 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:56.797129 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-12a423-2hvxd" podStartSLOduration=2.797111535 podStartE2EDuration="2.797111535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:13:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:13:56.794739686 +0000 UTC m=+2072.020059476" watchObservedRunningTime="2026-11-03 21:13:56.797111535 +0000 UTC m=+2072.022431326" Nov 03 21:13:56.897036 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:56.896997 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eff346af-88ef-494d-8505-c8546c5c6f4e" path="/var/lib/kubelet/pods/eff346af-88ef-494d-8505-c8546c5c6f4e/volumes" Nov 03 21:13:56.907818 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:56.907784 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-01c7a3-55bcd6997b-k8qhc"] Nov 03 21:13:56.911958 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:56.911927 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-01c7a3-55bcd6997b-k8qhc"] Nov 03 21:13:57.173107 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:57.173071 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44"] Nov 03 21:13:57.173569 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:57.173322 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44" podUID="4dbbf182-5d1d-4a54-8f71-bb58afa654e1" containerName="istio-proxy" containerID="cri-o://438f8ea599856ffa9ea59ba3f14e7c6de51041f43281cdf9c22cabe13785fd3f" gracePeriod=30 Nov 03 21:13:57.176251 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:57.176215 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-54764"] Nov 03 21:13:57.176452 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:57.176412 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-54764" podUID="5c9106f1-54e4-4c54-bba9-938e47117a27" containerName="istio-proxy" containerID="cri-o://bd80f226c3b8f9ec219a2624fb20a6a867fb13a000a19135a95b550af3da72ca" gracePeriod=30 Nov 03 21:13:57.504102 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:57.504017 4305 generic.go:356] "Generic (PLEG): container finished" podID="9b488427-d78e-41c3-848d-b5cf765a1e83" containerID="bb2a68447ca969d2a8aa6fd28ff9cd774d3af40c41d10c351a099fb5d20fe7bc" exitCode=0 Nov 03 21:13:57.504279 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:57.504104 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-12a423-2hvxd" event={"ID":"9b488427-d78e-41c3-848d-b5cf765a1e83","Type":"ContainerDied","Data":"bb2a68447ca969d2a8aa6fd28ff9cd774d3af40c41d10c351a099fb5d20fe7bc"} Nov 03 21:13:57.971672 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:57.971626 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-01c7a3-5x77g"] Nov 03 21:13:57.975117 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:57.975092 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-01c7a3-5x77g"] Nov 03 21:13:58.688269 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:58.688240 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-12a423-2hvxd" Nov 03 21:13:58.778300 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:58.778214 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b488427-d78e-41c3-848d-b5cf765a1e83-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b488427-d78e-41c3-848d-b5cf765a1e83-maas-api-openshift-ca\") pod \"9b488427-d78e-41c3-848d-b5cf765a1e83\" (UID: \"9b488427-d78e-41c3-848d-b5cf765a1e83\") " Nov 03 21:13:58.778496 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:58.778345 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b488427-d78e-41c3-848d-b5cf765a1e83-kube-api-access-2q7qs\" (UniqueName: \"kubernetes.io/projected/9b488427-d78e-41c3-848d-b5cf765a1e83-kube-api-access-2q7qs\") pod \"9b488427-d78e-41c3-848d-b5cf765a1e83\" (UID: \"9b488427-d78e-41c3-848d-b5cf765a1e83\") " Nov 03 21:13:58.778496 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:58.778376 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b488427-d78e-41c3-848d-b5cf765a1e83-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b488427-d78e-41c3-848d-b5cf765a1e83-maas-api-xks-ca\") pod \"9b488427-d78e-41c3-848d-b5cf765a1e83\" (UID: \"9b488427-d78e-41c3-848d-b5cf765a1e83\") " Nov 03 21:13:58.778704 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:58.778672 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b488427-d78e-41c3-848d-b5cf765a1e83-maas-api-openshift-ca" pod "9b488427-d78e-41c3-848d-b5cf765a1e83" (UID: "9b488427-d78e-41c3-848d-b5cf765a1e83"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:13:58.778870 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:58.778854 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b488427-d78e-41c3-848d-b5cf765a1e83-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:58.780841 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:58.780817 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b488427-d78e-41c3-848d-b5cf765a1e83-maas-api-xks-ca" pod "9b488427-d78e-41c3-848d-b5cf765a1e83" (UID: "9b488427-d78e-41c3-848d-b5cf765a1e83"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:13:58.780986 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:58.780841 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b488427-d78e-41c3-848d-b5cf765a1e83-kube-api-access-2q7qs" pod "9b488427-d78e-41c3-848d-b5cf765a1e83" (UID: "9b488427-d78e-41c3-848d-b5cf765a1e83"). InnerVolumeSpecName "kube-api-access-2q7qs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:13:58.879841 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:58.879804 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q7qs\" (UniqueName: \"kubernetes.io/projected/9b488427-d78e-41c3-848d-b5cf765a1e83-kube-api-access-2q7qs\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:58.879990 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:58.879846 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b488427-d78e-41c3-848d-b5cf765a1e83-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:58.900463 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:58.899725 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bf47f9f-ebc7-49ea-ae69-74712e627abb" path="/var/lib/kubelet/pods/7bf47f9f-ebc7-49ea-ae69-74712e627abb/volumes" Nov 03 21:13:58.900463 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:58.900339 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fb81be4-4690-45e1-947f-2de9f107d4d7" path="/var/lib/kubelet/pods/7fb81be4-4690-45e1-947f-2de9f107d4d7/volumes" Nov 03 21:13:59.117488 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:59.117461 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs" Nov 03 21:13:59.136689 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:59.136651 4305 patch_prober.go:29] interesting pod/e2e-worker-w2-862138-openshift-default-bc56c55fc-54764 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:13:59.136853 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:59.136703 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-54764" podUID="5c9106f1-54e4-4c54-bba9-938e47117a27" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:13:59.183466 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:59.183418 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4063115-7846-42ea-8bea-4684c27b132a-kube-api-access-qhr9r\" (UniqueName: \"kubernetes.io/projected/e4063115-7846-42ea-8bea-4684c27b132a-kube-api-access-qhr9r\") pod \"e4063115-7846-42ea-8bea-4684c27b132a\" (UID: \"e4063115-7846-42ea-8bea-4684c27b132a\") " Nov 03 21:13:59.183640 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:59.183495 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4063115-7846-42ea-8bea-4684c27b132a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4063115-7846-42ea-8bea-4684c27b132a-workload-certs\") pod \"e4063115-7846-42ea-8bea-4684c27b132a\" (UID: \"e4063115-7846-42ea-8bea-4684c27b132a\") " Nov 03 21:13:59.183640 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:59.183535 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4063115-7846-42ea-8bea-4684c27b132a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4063115-7846-42ea-8bea-4684c27b132a-workload-socket\") pod \"e4063115-7846-42ea-8bea-4684c27b132a\" (UID: \"e4063115-7846-42ea-8bea-4684c27b132a\") " Nov 03 21:13:59.183640 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:59.183557 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4063115-7846-42ea-8bea-4684c27b132a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4063115-7846-42ea-8bea-4684c27b132a-istiod-ca-cert\") pod \"e4063115-7846-42ea-8bea-4684c27b132a\" (UID: \"e4063115-7846-42ea-8bea-4684c27b132a\") " Nov 03 21:13:59.183640 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:59.183626 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4063115-7846-42ea-8bea-4684c27b132a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4063115-7846-42ea-8bea-4684c27b132a-istio-data\") pod \"e4063115-7846-42ea-8bea-4684c27b132a\" (UID: \"e4063115-7846-42ea-8bea-4684c27b132a\") " Nov 03 21:13:59.183826 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:59.183677 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4063115-7846-42ea-8bea-4684c27b132a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4063115-7846-42ea-8bea-4684c27b132a-credential-socket\") pod \"e4063115-7846-42ea-8bea-4684c27b132a\" (UID: \"e4063115-7846-42ea-8bea-4684c27b132a\") " Nov 03 21:13:59.183826 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:59.183734 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4063115-7846-42ea-8bea-4684c27b132a-istio-token\" (UniqueName: \"kubernetes.io/projected/e4063115-7846-42ea-8bea-4684c27b132a-istio-token\") pod \"e4063115-7846-42ea-8bea-4684c27b132a\" (UID: \"e4063115-7846-42ea-8bea-4684c27b132a\") " Nov 03 21:13:59.183826 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:59.183782 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e4063115-7846-42ea-8bea-4684c27b132a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4063115-7846-42ea-8bea-4684c27b132a-istio-podinfo\") pod \"e4063115-7846-42ea-8bea-4684c27b132a\" (UID: \"e4063115-7846-42ea-8bea-4684c27b132a\") " Nov 03 21:13:59.184003 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:59.183839 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4063115-7846-42ea-8bea-4684c27b132a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4063115-7846-42ea-8bea-4684c27b132a-istio-envoy\") pod \"e4063115-7846-42ea-8bea-4684c27b132a\" (UID: \"e4063115-7846-42ea-8bea-4684c27b132a\") " Nov 03 21:13:59.184003 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:59.183835 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4063115-7846-42ea-8bea-4684c27b132a-workload-socket" pod "e4063115-7846-42ea-8bea-4684c27b132a" (UID: "e4063115-7846-42ea-8bea-4684c27b132a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:13:59.184003 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:59.183846 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4063115-7846-42ea-8bea-4684c27b132a-workload-certs" pod "e4063115-7846-42ea-8bea-4684c27b132a" (UID: "e4063115-7846-42ea-8bea-4684c27b132a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:13:59.184003 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:59.183937 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4063115-7846-42ea-8bea-4684c27b132a-istio-data" pod "e4063115-7846-42ea-8bea-4684c27b132a" (UID: "e4063115-7846-42ea-8bea-4684c27b132a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:13:59.184263 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:59.184047 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4063115-7846-42ea-8bea-4684c27b132a-credential-socket" pod "e4063115-7846-42ea-8bea-4684c27b132a" (UID: "e4063115-7846-42ea-8bea-4684c27b132a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:13:59.184263 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:59.184063 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4063115-7846-42ea-8bea-4684c27b132a-istiod-ca-cert" pod "e4063115-7846-42ea-8bea-4684c27b132a" (UID: "e4063115-7846-42ea-8bea-4684c27b132a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:13:59.184727 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:59.184624 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4063115-7846-42ea-8bea-4684c27b132a-workload-certs\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:59.184727 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:59.184655 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4063115-7846-42ea-8bea-4684c27b132a-workload-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:59.184727 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:59.184671 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4063115-7846-42ea-8bea-4684c27b132a-istiod-ca-cert\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:59.184727 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:59.184686 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4063115-7846-42ea-8bea-4684c27b132a-istio-data\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:59.184727 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:59.184701 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4063115-7846-42ea-8bea-4684c27b132a-credential-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:59.186028 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:59.186006 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4063115-7846-42ea-8bea-4684c27b132a-kube-api-access-qhr9r" pod "e4063115-7846-42ea-8bea-4684c27b132a" (UID: "e4063115-7846-42ea-8bea-4684c27b132a"). InnerVolumeSpecName "kube-api-access-qhr9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:13:59.186128 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:59.186075 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4063115-7846-42ea-8bea-4684c27b132a-istio-envoy" pod "e4063115-7846-42ea-8bea-4684c27b132a" (UID: "e4063115-7846-42ea-8bea-4684c27b132a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:13:59.186167 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:59.186124 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4063115-7846-42ea-8bea-4684c27b132a-istio-token" pod "e4063115-7846-42ea-8bea-4684c27b132a" (UID: "e4063115-7846-42ea-8bea-4684c27b132a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:13:59.186200 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:59.186178 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e4063115-7846-42ea-8bea-4684c27b132a-istio-podinfo" pod "e4063115-7846-42ea-8bea-4684c27b132a" (UID: "e4063115-7846-42ea-8bea-4684c27b132a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:13:59.286116 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:59.286024 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhr9r\" (UniqueName: \"kubernetes.io/projected/e4063115-7846-42ea-8bea-4684c27b132a-kube-api-access-qhr9r\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:59.286116 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:59.286068 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4063115-7846-42ea-8bea-4684c27b132a-istio-token\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:59.286116 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:59.286078 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4063115-7846-42ea-8bea-4684c27b132a-istio-podinfo\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:59.286116 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:59.286088 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4063115-7846-42ea-8bea-4684c27b132a-istio-envoy\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:13:59.523578 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:59.523542 4305 generic.go:356] "Generic (PLEG): container finished" podID="e4063115-7846-42ea-8bea-4684c27b132a" containerID="9795139d2fdc6878b2ca6a18c3b3908b403cd54e700ce49c53546f48b6ace684" exitCode=0 Nov 03 21:13:59.523759 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:59.523600 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs" Nov 03 21:13:59.523759 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:59.523619 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs" event={"ID":"e4063115-7846-42ea-8bea-4684c27b132a","Type":"ContainerDied","Data":"9795139d2fdc6878b2ca6a18c3b3908b403cd54e700ce49c53546f48b6ace684"} Nov 03 21:13:59.523759 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:59.523683 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs" event={"ID":"e4063115-7846-42ea-8bea-4684c27b132a","Type":"ContainerDied","Data":"de44b3e25aa41366c6d35d7b8f91d8dc03602daf7b0eb1a23895a8a87d16d8e5"} Nov 03 21:13:59.523759 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:59.523712 4305 scope.go:122] "RemoveContainer" containerID="9795139d2fdc6878b2ca6a18c3b3908b403cd54e700ce49c53546f48b6ace684" Nov 03 21:13:59.525233 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:59.525208 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-12a423-2hvxd" event={"ID":"9b488427-d78e-41c3-848d-b5cf765a1e83","Type":"ContainerDied","Data":"56bea4722d022ca53edbffd328e53cae91fab35c927c7f75d2bc0bc0f7149c52"} Nov 03 21:13:59.525304 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:59.525243 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-12a423-2hvxd" Nov 03 21:13:59.525372 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:59.525260 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56bea4722d022ca53edbffd328e53cae91fab35c927c7f75d2bc0bc0f7149c52" Nov 03 21:13:59.535364 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:59.535344 4305 scope.go:122] "RemoveContainer" containerID="18c8bfc24d304e2064190791989c8695d18a0eaf3e807aef35efaa067e6aa293" Nov 03 21:13:59.545880 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:59.545862 4305 scope.go:122] "RemoveContainer" containerID="9795139d2fdc6878b2ca6a18c3b3908b403cd54e700ce49c53546f48b6ace684" Nov 03 21:13:59.546168 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:13:59.546150 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9795139d2fdc6878b2ca6a18c3b3908b403cd54e700ce49c53546f48b6ace684\": container with ID starting with 9795139d2fdc6878b2ca6a18c3b3908b403cd54e700ce49c53546f48b6ace684 not found: ID does not exist" containerID="9795139d2fdc6878b2ca6a18c3b3908b403cd54e700ce49c53546f48b6ace684" Nov 03 21:13:59.546237 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:59.546177 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9795139d2fdc6878b2ca6a18c3b3908b403cd54e700ce49c53546f48b6ace684"} err="failed to get container status \"9795139d2fdc6878b2ca6a18c3b3908b403cd54e700ce49c53546f48b6ace684\": rpc error: code = NotFound desc = could not find container \"9795139d2fdc6878b2ca6a18c3b3908b403cd54e700ce49c53546f48b6ace684\": container with ID starting with 9795139d2fdc6878b2ca6a18c3b3908b403cd54e700ce49c53546f48b6ace684 not found: ID does not exist" Nov 03 21:13:59.546237 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:59.546194 4305 scope.go:122] "RemoveContainer" containerID="18c8bfc24d304e2064190791989c8695d18a0eaf3e807aef35efaa067e6aa293" Nov 03 21:13:59.546447 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:13:59.546417 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18c8bfc24d304e2064190791989c8695d18a0eaf3e807aef35efaa067e6aa293\": container with ID starting with 18c8bfc24d304e2064190791989c8695d18a0eaf3e807aef35efaa067e6aa293 not found: ID does not exist" containerID="18c8bfc24d304e2064190791989c8695d18a0eaf3e807aef35efaa067e6aa293" Nov 03 21:13:59.546487 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:59.546452 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18c8bfc24d304e2064190791989c8695d18a0eaf3e807aef35efaa067e6aa293"} err="failed to get container status \"18c8bfc24d304e2064190791989c8695d18a0eaf3e807aef35efaa067e6aa293\": rpc error: code = NotFound desc = could not find container \"18c8bfc24d304e2064190791989c8695d18a0eaf3e807aef35efaa067e6aa293\": container with ID starting with 18c8bfc24d304e2064190791989c8695d18a0eaf3e807aef35efaa067e6aa293 not found: ID does not exist" Nov 03 21:13:59.713056 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:59.713019 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs"] Nov 03 21:13:59.762675 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:59.762646 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-1e5494-openshift-default-56b5774bcc-qf7cs"] Nov 03 21:13:59.905065 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:13:59.905026 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-e22abf-c94f8f5d8-n4hfh"] Nov 03 21:14:00.895826 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:00.895790 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4063115-7846-42ea-8bea-4684c27b132a" path="/var/lib/kubelet/pods/e4063115-7846-42ea-8bea-4684c27b132a/volumes" Nov 03 21:14:02.058000 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.057966 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb6523bb-dd4qf"] Nov 03 21:14:02.108140 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.108098 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-346e532f-lztv8"] Nov 03 21:14:02.112522 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.112492 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb6523bb-dd4qf"] Nov 03 21:14:02.117584 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.117553 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-346e532f-lztv8"] Nov 03 21:14:02.447478 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.447423 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3a7b6c5-4rgn4"] Nov 03 21:14:02.447987 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.447971 4305 cpu_manager.go:414] "containerMap: removing container" podUID="e4063115-7846-42ea-8bea-4684c27b132a" containerName="istio-proxy" Nov 03 21:14:02.448072 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.447988 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4063115-7846-42ea-8bea-4684c27b132a" containerName="istio-proxy" Nov 03 21:14:02.448072 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.448000 4305 cpu_manager.go:414] "containerMap: removing container" podUID="7fb81be4-4690-45e1-947f-2de9f107d4d7" containerName="payload-processing" Nov 03 21:14:02.448072 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.448005 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fb81be4-4690-45e1-947f-2de9f107d4d7" containerName="payload-processing" Nov 03 21:14:02.448072 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.448023 4305 cpu_manager.go:414] "containerMap: removing container" podUID="eff346af-88ef-494d-8505-c8546c5c6f4e" containerName="payload-pre-processing" Nov 03 21:14:02.448072 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.448029 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eff346af-88ef-494d-8505-c8546c5c6f4e" containerName="payload-pre-processing" Nov 03 21:14:02.448072 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.448048 4305 cpu_manager.go:414] "containerMap: removing container" podUID="9b488427-d78e-41c3-848d-b5cf765a1e83" containerName="revoke-keys" Nov 03 21:14:02.448072 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.448053 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b488427-d78e-41c3-848d-b5cf765a1e83" containerName="revoke-keys" Nov 03 21:14:02.448072 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.448070 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-54764" Nov 03 21:14:02.448525 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.448272 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4063115-7846-42ea-8bea-4684c27b132a" containerName="istio-proxy" Nov 03 21:14:02.448525 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.448291 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4063115-7846-42ea-8bea-4684c27b132a" containerName="istio-proxy" Nov 03 21:14:02.448525 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.448303 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fb81be4-4690-45e1-947f-2de9f107d4d7" containerName="payload-processing" Nov 03 21:14:02.448525 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.448318 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="eff346af-88ef-494d-8505-c8546c5c6f4e" containerName="payload-pre-processing" Nov 03 21:14:02.448525 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.448336 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b488427-d78e-41c3-848d-b5cf765a1e83" containerName="revoke-keys" Nov 03 21:14:02.452292 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.451992 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a7b6c5-4rgn4" Nov 03 21:14:02.481637 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.481613 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44" Nov 03 21:14:02.509595 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.509562 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ae29793-n5kjl"] Nov 03 21:14:02.510271 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.510252 4305 cpu_manager.go:414] "containerMap: removing container" podUID="5c9106f1-54e4-4c54-bba9-938e47117a27" containerName="istio-proxy" Nov 03 21:14:02.510349 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.510273 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c9106f1-54e4-4c54-bba9-938e47117a27" containerName="istio-proxy" Nov 03 21:14:02.510349 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.510320 4305 cpu_manager.go:414] "containerMap: removing container" podUID="4dbbf182-5d1d-4a54-8f71-bb58afa654e1" containerName="istio-proxy" Nov 03 21:14:02.510349 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.510328 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dbbf182-5d1d-4a54-8f71-bb58afa654e1" containerName="istio-proxy" Nov 03 21:14:02.510349 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.510343 4305 cpu_manager.go:414] "containerMap: removing container" podUID="e4063115-7846-42ea-8bea-4684c27b132a" containerName="istio-proxy" Nov 03 21:14:02.510349 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.510349 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4063115-7846-42ea-8bea-4684c27b132a" containerName="istio-proxy" Nov 03 21:14:02.510664 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.510476 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dbbf182-5d1d-4a54-8f71-bb58afa654e1" containerName="istio-proxy" Nov 03 21:14:02.510664 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.510492 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c9106f1-54e4-4c54-bba9-938e47117a27" containerName="istio-proxy" Nov 03 21:14:02.513834 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.513813 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ae29793-n5kjl" Nov 03 21:14:02.531353 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.531321 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c9106f1-54e4-4c54-bba9-938e47117a27-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c9106f1-54e4-4c54-bba9-938e47117a27-workload-certs\") pod \"5c9106f1-54e4-4c54-bba9-938e47117a27\" (UID: \"5c9106f1-54e4-4c54-bba9-938e47117a27\") " Nov 03 21:14:02.531577 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.531377 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c9106f1-54e4-4c54-bba9-938e47117a27-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c9106f1-54e4-4c54-bba9-938e47117a27-istio-data\") pod \"5c9106f1-54e4-4c54-bba9-938e47117a27\" (UID: \"5c9106f1-54e4-4c54-bba9-938e47117a27\") " Nov 03 21:14:02.531577 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.531419 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c9106f1-54e4-4c54-bba9-938e47117a27-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c9106f1-54e4-4c54-bba9-938e47117a27-workload-socket\") pod \"5c9106f1-54e4-4c54-bba9-938e47117a27\" (UID: \"5c9106f1-54e4-4c54-bba9-938e47117a27\") " Nov 03 21:14:02.531577 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.531504 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c9106f1-54e4-4c54-bba9-938e47117a27-istio-token\" (UniqueName: \"kubernetes.io/projected/5c9106f1-54e4-4c54-bba9-938e47117a27-istio-token\") pod \"5c9106f1-54e4-4c54-bba9-938e47117a27\" (UID: \"5c9106f1-54e4-4c54-bba9-938e47117a27\") " Nov 03 21:14:02.531577 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.531541 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c9106f1-54e4-4c54-bba9-938e47117a27-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c9106f1-54e4-4c54-bba9-938e47117a27-istio-podinfo\") pod \"5c9106f1-54e4-4c54-bba9-938e47117a27\" (UID: \"5c9106f1-54e4-4c54-bba9-938e47117a27\") " Nov 03 21:14:02.531815 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.531621 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c9106f1-54e4-4c54-bba9-938e47117a27-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c9106f1-54e4-4c54-bba9-938e47117a27-istio-envoy\") pod \"5c9106f1-54e4-4c54-bba9-938e47117a27\" (UID: \"5c9106f1-54e4-4c54-bba9-938e47117a27\") " Nov 03 21:14:02.531815 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.531623 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c9106f1-54e4-4c54-bba9-938e47117a27-workload-certs" pod "5c9106f1-54e4-4c54-bba9-938e47117a27" (UID: "5c9106f1-54e4-4c54-bba9-938e47117a27"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:02.531815 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.531654 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c9106f1-54e4-4c54-bba9-938e47117a27-istio-data" pod "5c9106f1-54e4-4c54-bba9-938e47117a27" (UID: "5c9106f1-54e4-4c54-bba9-938e47117a27"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:02.531815 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.531732 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c9106f1-54e4-4c54-bba9-938e47117a27-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c9106f1-54e4-4c54-bba9-938e47117a27-credential-socket\") pod \"5c9106f1-54e4-4c54-bba9-938e47117a27\" (UID: \"5c9106f1-54e4-4c54-bba9-938e47117a27\") " Nov 03 21:14:02.531815 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.531805 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c9106f1-54e4-4c54-bba9-938e47117a27-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c9106f1-54e4-4c54-bba9-938e47117a27-istiod-ca-cert\") pod \"5c9106f1-54e4-4c54-bba9-938e47117a27\" (UID: \"5c9106f1-54e4-4c54-bba9-938e47117a27\") " Nov 03 21:14:02.532089 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.531852 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c9106f1-54e4-4c54-bba9-938e47117a27-kube-api-access-tjqsd\" (UniqueName: \"kubernetes.io/projected/5c9106f1-54e4-4c54-bba9-938e47117a27-kube-api-access-tjqsd\") pod \"5c9106f1-54e4-4c54-bba9-938e47117a27\" (UID: \"5c9106f1-54e4-4c54-bba9-938e47117a27\") " Nov 03 21:14:02.532089 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.532042 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de2f5dab-53d4-4e0d-be0e-e71aa193705a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3a7b6c5-4rgn4\" (UID: \"de2f5dab-53d4-4e0d-be0e-e71aa193705a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a7b6c5-4rgn4" Nov 03 21:14:02.532201 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.532108 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de2f5dab-53d4-4e0d-be0e-e71aa193705a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3a7b6c5-4rgn4\" (UID: \"de2f5dab-53d4-4e0d-be0e-e71aa193705a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a7b6c5-4rgn4" Nov 03 21:14:02.532201 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.532183 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c9106f1-54e4-4c54-bba9-938e47117a27-credential-socket" pod "5c9106f1-54e4-4c54-bba9-938e47117a27" (UID: "5c9106f1-54e4-4c54-bba9-938e47117a27"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:02.533505 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.532597 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh6jc\" (UniqueName: \"kubernetes.io/projected/de2f5dab-53d4-4e0d-be0e-e71aa193705a-kube-api-access-mh6jc\") pod \"maas-api-revoke-sub-f3a7b6c5-4rgn4\" (UID: \"de2f5dab-53d4-4e0d-be0e-e71aa193705a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a7b6c5-4rgn4" Nov 03 21:14:02.533505 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.532765 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c9106f1-54e4-4c54-bba9-938e47117a27-workload-certs\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:02.533505 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.532784 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c9106f1-54e4-4c54-bba9-938e47117a27-istio-data\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:02.533505 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.532800 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c9106f1-54e4-4c54-bba9-938e47117a27-credential-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:02.533505 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.533213 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c9106f1-54e4-4c54-bba9-938e47117a27-istiod-ca-cert" pod "5c9106f1-54e4-4c54-bba9-938e47117a27" (UID: "5c9106f1-54e4-4c54-bba9-938e47117a27"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:14:02.534129 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.534089 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c9106f1-54e4-4c54-bba9-938e47117a27-workload-socket" pod "5c9106f1-54e4-4c54-bba9-938e47117a27" (UID: "5c9106f1-54e4-4c54-bba9-938e47117a27"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:02.534992 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.534964 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c9106f1-54e4-4c54-bba9-938e47117a27-kube-api-access-tjqsd" pod "5c9106f1-54e4-4c54-bba9-938e47117a27" (UID: "5c9106f1-54e4-4c54-bba9-938e47117a27"). InnerVolumeSpecName "kube-api-access-tjqsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:14:02.538404 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.538315 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c9106f1-54e4-4c54-bba9-938e47117a27-istio-token" pod "5c9106f1-54e4-4c54-bba9-938e47117a27" (UID: "5c9106f1-54e4-4c54-bba9-938e47117a27"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:14:02.538404 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.538398 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c9106f1-54e4-4c54-bba9-938e47117a27-istio-podinfo" pod "5c9106f1-54e4-4c54-bba9-938e47117a27" (UID: "5c9106f1-54e4-4c54-bba9-938e47117a27"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:14:02.539828 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.539807 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c9106f1-54e4-4c54-bba9-938e47117a27-istio-envoy" pod "5c9106f1-54e4-4c54-bba9-938e47117a27" (UID: "5c9106f1-54e4-4c54-bba9-938e47117a27"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:02.550551 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.550519 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:14:02.550551 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.550549 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:14:02.550766 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.550546 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77vtd\"" Nov 03 21:14:02.550766 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.550522 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:14:02.554426 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.554392 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3a7b6c5-4rgn4"] Nov 03 21:14:02.562196 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.562162 4305 generic.go:356] "Generic (PLEG): container finished" podID="4dbbf182-5d1d-4a54-8f71-bb58afa654e1" containerID="438f8ea599856ffa9ea59ba3f14e7c6de51041f43281cdf9c22cabe13785fd3f" exitCode=0 Nov 03 21:14:02.562319 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.562234 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44" Nov 03 21:14:02.562319 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.562257 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44" event={"ID":"4dbbf182-5d1d-4a54-8f71-bb58afa654e1","Type":"ContainerDied","Data":"438f8ea599856ffa9ea59ba3f14e7c6de51041f43281cdf9c22cabe13785fd3f"} Nov 03 21:14:02.562319 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.562310 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44" event={"ID":"4dbbf182-5d1d-4a54-8f71-bb58afa654e1","Type":"ContainerDied","Data":"2c5c72f54bd405be2f1f818e40d4610387d8b924e20b445be6a93c66ba266eb5"} Nov 03 21:14:02.562541 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.562340 4305 scope.go:122] "RemoveContainer" containerID="438f8ea599856ffa9ea59ba3f14e7c6de51041f43281cdf9c22cabe13785fd3f" Nov 03 21:14:02.564142 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.564090 4305 generic.go:356] "Generic (PLEG): container finished" podID="5c9106f1-54e4-4c54-bba9-938e47117a27" containerID="bd80f226c3b8f9ec219a2624fb20a6a867fb13a000a19135a95b550af3da72ca" exitCode=0 Nov 03 21:14:02.564142 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.564150 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-54764" Nov 03 21:14:02.564507 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.564173 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-54764" event={"ID":"5c9106f1-54e4-4c54-bba9-938e47117a27","Type":"ContainerDied","Data":"bd80f226c3b8f9ec219a2624fb20a6a867fb13a000a19135a95b550af3da72ca"} Nov 03 21:14:02.564507 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.564236 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-54764" event={"ID":"5c9106f1-54e4-4c54-bba9-938e47117a27","Type":"ContainerDied","Data":"304393eb01df074cafe626c70bdcd44b35066904f8515ee2d7d7430e2bdaf0f2"} Nov 03 21:14:02.576879 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.576829 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ae29793-n5kjl"] Nov 03 21:14:02.579048 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.579027 4305 scope.go:122] "RemoveContainer" containerID="438f8ea599856ffa9ea59ba3f14e7c6de51041f43281cdf9c22cabe13785fd3f" Nov 03 21:14:02.579454 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:14:02.579397 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"438f8ea599856ffa9ea59ba3f14e7c6de51041f43281cdf9c22cabe13785fd3f\": container with ID starting with 438f8ea599856ffa9ea59ba3f14e7c6de51041f43281cdf9c22cabe13785fd3f not found: ID does not exist" containerID="438f8ea599856ffa9ea59ba3f14e7c6de51041f43281cdf9c22cabe13785fd3f" Nov 03 21:14:02.579592 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.579481 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"438f8ea599856ffa9ea59ba3f14e7c6de51041f43281cdf9c22cabe13785fd3f"} err="failed to get container status \"438f8ea599856ffa9ea59ba3f14e7c6de51041f43281cdf9c22cabe13785fd3f\": rpc error: code = NotFound desc = could not find container \"438f8ea599856ffa9ea59ba3f14e7c6de51041f43281cdf9c22cabe13785fd3f\": container with ID starting with 438f8ea599856ffa9ea59ba3f14e7c6de51041f43281cdf9c22cabe13785fd3f not found: ID does not exist" Nov 03 21:14:02.579592 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.579512 4305 scope.go:122] "RemoveContainer" containerID="bd80f226c3b8f9ec219a2624fb20a6a867fb13a000a19135a95b550af3da72ca" Nov 03 21:14:02.595378 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.595285 4305 scope.go:122] "RemoveContainer" containerID="bd80f226c3b8f9ec219a2624fb20a6a867fb13a000a19135a95b550af3da72ca" Nov 03 21:14:02.596162 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:14:02.596114 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd80f226c3b8f9ec219a2624fb20a6a867fb13a000a19135a95b550af3da72ca\": container with ID starting with bd80f226c3b8f9ec219a2624fb20a6a867fb13a000a19135a95b550af3da72ca not found: ID does not exist" containerID="bd80f226c3b8f9ec219a2624fb20a6a867fb13a000a19135a95b550af3da72ca" Nov 03 21:14:02.596339 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.596158 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd80f226c3b8f9ec219a2624fb20a6a867fb13a000a19135a95b550af3da72ca"} err="failed to get container status \"bd80f226c3b8f9ec219a2624fb20a6a867fb13a000a19135a95b550af3da72ca\": rpc error: code = NotFound desc = could not find container \"bd80f226c3b8f9ec219a2624fb20a6a867fb13a000a19135a95b550af3da72ca\": container with ID starting with bd80f226c3b8f9ec219a2624fb20a6a867fb13a000a19135a95b550af3da72ca not found: ID does not exist" Nov 03 21:14:02.633414 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.633359 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dbbf182-5d1d-4a54-8f71-bb58afa654e1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dbbf182-5d1d-4a54-8f71-bb58afa654e1-workload-certs\") pod \"4dbbf182-5d1d-4a54-8f71-bb58afa654e1\" (UID: \"4dbbf182-5d1d-4a54-8f71-bb58afa654e1\") " Nov 03 21:14:02.633619 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.633571 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dbbf182-5d1d-4a54-8f71-bb58afa654e1-istio-token\" (UniqueName: \"kubernetes.io/projected/4dbbf182-5d1d-4a54-8f71-bb58afa654e1-istio-token\") pod \"4dbbf182-5d1d-4a54-8f71-bb58afa654e1\" (UID: \"4dbbf182-5d1d-4a54-8f71-bb58afa654e1\") " Nov 03 21:14:02.633619 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.633605 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4dbbf182-5d1d-4a54-8f71-bb58afa654e1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dbbf182-5d1d-4a54-8f71-bb58afa654e1-istio-podinfo\") pod \"4dbbf182-5d1d-4a54-8f71-bb58afa654e1\" (UID: \"4dbbf182-5d1d-4a54-8f71-bb58afa654e1\") " Nov 03 21:14:02.633722 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.633638 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dbbf182-5d1d-4a54-8f71-bb58afa654e1-workload-certs" pod "4dbbf182-5d1d-4a54-8f71-bb58afa654e1" (UID: "4dbbf182-5d1d-4a54-8f71-bb58afa654e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:02.633722 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.633675 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dbbf182-5d1d-4a54-8f71-bb58afa654e1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dbbf182-5d1d-4a54-8f71-bb58afa654e1-credential-socket\") pod \"4dbbf182-5d1d-4a54-8f71-bb58afa654e1\" (UID: \"4dbbf182-5d1d-4a54-8f71-bb58afa654e1\") " Nov 03 21:14:02.633722 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.633709 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dbbf182-5d1d-4a54-8f71-bb58afa654e1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dbbf182-5d1d-4a54-8f71-bb58afa654e1-workload-socket\") pod \"4dbbf182-5d1d-4a54-8f71-bb58afa654e1\" (UID: \"4dbbf182-5d1d-4a54-8f71-bb58afa654e1\") " Nov 03 21:14:02.633883 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.633831 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dbbf182-5d1d-4a54-8f71-bb58afa654e1-kube-api-access-x4b8z\" (UniqueName: \"kubernetes.io/projected/4dbbf182-5d1d-4a54-8f71-bb58afa654e1-kube-api-access-x4b8z\") pod \"4dbbf182-5d1d-4a54-8f71-bb58afa654e1\" (UID: \"4dbbf182-5d1d-4a54-8f71-bb58afa654e1\") " Nov 03 21:14:02.633946 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.633888 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dbbf182-5d1d-4a54-8f71-bb58afa654e1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dbbf182-5d1d-4a54-8f71-bb58afa654e1-istio-envoy\") pod \"4dbbf182-5d1d-4a54-8f71-bb58afa654e1\" (UID: \"4dbbf182-5d1d-4a54-8f71-bb58afa654e1\") " Nov 03 21:14:02.633946 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.633931 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dbbf182-5d1d-4a54-8f71-bb58afa654e1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dbbf182-5d1d-4a54-8f71-bb58afa654e1-istiod-ca-cert\") pod \"4dbbf182-5d1d-4a54-8f71-bb58afa654e1\" (UID: \"4dbbf182-5d1d-4a54-8f71-bb58afa654e1\") " Nov 03 21:14:02.634132 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.634023 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dbbf182-5d1d-4a54-8f71-bb58afa654e1-credential-socket" pod "4dbbf182-5d1d-4a54-8f71-bb58afa654e1" (UID: "4dbbf182-5d1d-4a54-8f71-bb58afa654e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:02.634132 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.634036 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dbbf182-5d1d-4a54-8f71-bb58afa654e1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dbbf182-5d1d-4a54-8f71-bb58afa654e1-istio-data\") pod \"4dbbf182-5d1d-4a54-8f71-bb58afa654e1\" (UID: \"4dbbf182-5d1d-4a54-8f71-bb58afa654e1\") " Nov 03 21:14:02.634132 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.634040 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dbbf182-5d1d-4a54-8f71-bb58afa654e1-workload-socket" pod "4dbbf182-5d1d-4a54-8f71-bb58afa654e1" (UID: "4dbbf182-5d1d-4a54-8f71-bb58afa654e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:02.634294 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.634269 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh6jc\" (UniqueName: \"kubernetes.io/projected/de2f5dab-53d4-4e0d-be0e-e71aa193705a-kube-api-access-mh6jc\") pod \"maas-api-revoke-sub-f3a7b6c5-4rgn4\" (UID: \"de2f5dab-53d4-4e0d-be0e-e71aa193705a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a7b6c5-4rgn4" Nov 03 21:14:02.634380 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.634353 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dbbf182-5d1d-4a54-8f71-bb58afa654e1-istio-data" pod "4dbbf182-5d1d-4a54-8f71-bb58afa654e1" (UID: "4dbbf182-5d1d-4a54-8f71-bb58afa654e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:02.634486 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.634361 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dbbf182-5d1d-4a54-8f71-bb58afa654e1-istiod-ca-cert" pod "4dbbf182-5d1d-4a54-8f71-bb58afa654e1" (UID: "4dbbf182-5d1d-4a54-8f71-bb58afa654e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:14:02.634486 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.634402 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de2f5dab-53d4-4e0d-be0e-e71aa193705a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3a7b6c5-4rgn4\" (UID: \"de2f5dab-53d4-4e0d-be0e-e71aa193705a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a7b6c5-4rgn4" Nov 03 21:14:02.634486 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.634472 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de2f5dab-53d4-4e0d-be0e-e71aa193705a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3a7b6c5-4rgn4\" (UID: \"de2f5dab-53d4-4e0d-be0e-e71aa193705a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a7b6c5-4rgn4" Nov 03 21:14:02.634673 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.634529 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b44q5\" (UniqueName: \"kubernetes.io/projected/07f1475c-e840-466e-bc78-75665a1468ba-kube-api-access-b44q5\") pod \"maas-api-revoke-sub-6ae29793-n5kjl\" (UID: \"07f1475c-e840-466e-bc78-75665a1468ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ae29793-n5kjl" Nov 03 21:14:02.634673 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.634579 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07f1475c-e840-466e-bc78-75665a1468ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ae29793-n5kjl\" (UID: \"07f1475c-e840-466e-bc78-75665a1468ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ae29793-n5kjl" Nov 03 21:14:02.634673 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.634609 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07f1475c-e840-466e-bc78-75665a1468ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ae29793-n5kjl\" (UID: \"07f1475c-e840-466e-bc78-75665a1468ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ae29793-n5kjl" Nov 03 21:14:02.634908 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.634885 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c9106f1-54e4-4c54-bba9-938e47117a27-istiod-ca-cert\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:02.635115 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.634912 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dbbf182-5d1d-4a54-8f71-bb58afa654e1-istio-data\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:02.635115 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.634927 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjqsd\" (UniqueName: \"kubernetes.io/projected/5c9106f1-54e4-4c54-bba9-938e47117a27-kube-api-access-tjqsd\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:02.635115 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.634942 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dbbf182-5d1d-4a54-8f71-bb58afa654e1-workload-certs\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:02.635115 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.634957 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c9106f1-54e4-4c54-bba9-938e47117a27-workload-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:02.635115 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.634970 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c9106f1-54e4-4c54-bba9-938e47117a27-istio-token\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:02.635115 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.634985 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c9106f1-54e4-4c54-bba9-938e47117a27-istio-podinfo\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:02.635115 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.634999 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c9106f1-54e4-4c54-bba9-938e47117a27-istio-envoy\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:02.635115 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.635014 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dbbf182-5d1d-4a54-8f71-bb58afa654e1-credential-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:02.635587 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.635119 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dbbf182-5d1d-4a54-8f71-bb58afa654e1-workload-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:02.635587 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.635202 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dbbf182-5d1d-4a54-8f71-bb58afa654e1-istiod-ca-cert\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:02.635587 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.635202 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de2f5dab-53d4-4e0d-be0e-e71aa193705a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3a7b6c5-4rgn4\" (UID: \"de2f5dab-53d4-4e0d-be0e-e71aa193705a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a7b6c5-4rgn4" Nov 03 21:14:02.636897 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.636860 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4dbbf182-5d1d-4a54-8f71-bb58afa654e1-istio-podinfo" pod "4dbbf182-5d1d-4a54-8f71-bb58afa654e1" (UID: "4dbbf182-5d1d-4a54-8f71-bb58afa654e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:14:02.637199 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.637150 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dbbf182-5d1d-4a54-8f71-bb58afa654e1-istio-envoy" pod "4dbbf182-5d1d-4a54-8f71-bb58afa654e1" (UID: "4dbbf182-5d1d-4a54-8f71-bb58afa654e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:02.637387 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.637370 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dbbf182-5d1d-4a54-8f71-bb58afa654e1-kube-api-access-x4b8z" pod "4dbbf182-5d1d-4a54-8f71-bb58afa654e1" (UID: "4dbbf182-5d1d-4a54-8f71-bb58afa654e1"). InnerVolumeSpecName "kube-api-access-x4b8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:14:02.637552 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.637537 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dbbf182-5d1d-4a54-8f71-bb58afa654e1-istio-token" pod "4dbbf182-5d1d-4a54-8f71-bb58afa654e1" (UID: "4dbbf182-5d1d-4a54-8f71-bb58afa654e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:14:02.639222 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.639198 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de2f5dab-53d4-4e0d-be0e-e71aa193705a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3a7b6c5-4rgn4\" (UID: \"de2f5dab-53d4-4e0d-be0e-e71aa193705a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a7b6c5-4rgn4" Nov 03 21:14:02.695339 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.695300 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh6jc\" (UniqueName: \"kubernetes.io/projected/de2f5dab-53d4-4e0d-be0e-e71aa193705a-kube-api-access-mh6jc\") pod \"maas-api-revoke-sub-f3a7b6c5-4rgn4\" (UID: \"de2f5dab-53d4-4e0d-be0e-e71aa193705a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a7b6c5-4rgn4" Nov 03 21:14:02.735977 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.735935 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b44q5\" (UniqueName: \"kubernetes.io/projected/07f1475c-e840-466e-bc78-75665a1468ba-kube-api-access-b44q5\") pod \"maas-api-revoke-sub-6ae29793-n5kjl\" (UID: \"07f1475c-e840-466e-bc78-75665a1468ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ae29793-n5kjl" Nov 03 21:14:02.736185 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.735999 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07f1475c-e840-466e-bc78-75665a1468ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ae29793-n5kjl\" (UID: \"07f1475c-e840-466e-bc78-75665a1468ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ae29793-n5kjl" Nov 03 21:14:02.736185 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.736028 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07f1475c-e840-466e-bc78-75665a1468ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ae29793-n5kjl\" (UID: \"07f1475c-e840-466e-bc78-75665a1468ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ae29793-n5kjl" Nov 03 21:14:02.736185 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.736174 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dbbf182-5d1d-4a54-8f71-bb58afa654e1-istio-token\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:02.736330 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.736190 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dbbf182-5d1d-4a54-8f71-bb58afa654e1-istio-podinfo\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:02.736330 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.736204 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4b8z\" (UniqueName: \"kubernetes.io/projected/4dbbf182-5d1d-4a54-8f71-bb58afa654e1-kube-api-access-x4b8z\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:02.736330 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.736218 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dbbf182-5d1d-4a54-8f71-bb58afa654e1-istio-envoy\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:02.736690 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.736669 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07f1475c-e840-466e-bc78-75665a1468ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ae29793-n5kjl\" (UID: \"07f1475c-e840-466e-bc78-75665a1468ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ae29793-n5kjl" Nov 03 21:14:02.738586 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.738567 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07f1475c-e840-466e-bc78-75665a1468ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ae29793-n5kjl\" (UID: \"07f1475c-e840-466e-bc78-75665a1468ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ae29793-n5kjl" Nov 03 21:14:02.780557 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.780516 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a7b6c5-4rgn4" Nov 03 21:14:02.797688 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.797657 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b44q5\" (UniqueName: \"kubernetes.io/projected/07f1475c-e840-466e-bc78-75665a1468ba-kube-api-access-b44q5\") pod \"maas-api-revoke-sub-6ae29793-n5kjl\" (UID: \"07f1475c-e840-466e-bc78-75665a1468ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ae29793-n5kjl" Nov 03 21:14:02.831084 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.831045 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ae29793-n5kjl" Nov 03 21:14:02.897812 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.897657 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fb6d58d-39c4-4f67-876e-7000d5833d82" path="/var/lib/kubelet/pods/6fb6d58d-39c4-4f67-876e-7000d5833d82/volumes" Nov 03 21:14:02.900288 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:02.898162 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99b7150f-e466-4734-97db-cde9002e38b0" path="/var/lib/kubelet/pods/99b7150f-e466-4734-97db-cde9002e38b0/volumes" Nov 03 21:14:03.111186 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:03.111095 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44"] Nov 03 21:14:03.115715 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:03.115675 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-btf44"] Nov 03 21:14:03.160768 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:03.160741 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3a7b6c5-4rgn4"] Nov 03 21:14:03.161646 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:14:03.161617 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde2f5dab_53d4_4e0d_be0e_e71aa193705a.slice/crio-2ea3b1f97ced607252e4964dab4be31136e646690ec95d4086ed2bc478ef3234 WatchSource:0}: Error finding container 2ea3b1f97ced607252e4964dab4be31136e646690ec95d4086ed2bc478ef3234: Status 404 returned error can't find the container with id 2ea3b1f97ced607252e4964dab4be31136e646690ec95d4086ed2bc478ef3234 Nov 03 21:14:03.258428 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:03.258381 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-54764"] Nov 03 21:14:03.277393 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:03.277360 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-54764"] Nov 03 21:14:03.360823 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:03.360788 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ae29793-n5kjl"] Nov 03 21:14:03.365887 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:14:03.365853 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07f1475c_e840_466e_bc78_75665a1468ba.slice/crio-43aeb73d697f519bcf8f5ef814cc90c590b5ce51030de9477c100a4d948c78a4 WatchSource:0}: Error finding container 43aeb73d697f519bcf8f5ef814cc90c590b5ce51030de9477c100a4d948c78a4: Status 404 returned error can't find the container with id 43aeb73d697f519bcf8f5ef814cc90c590b5ce51030de9477c100a4d948c78a4 Nov 03 21:14:03.573417 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:03.573378 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a7b6c5-4rgn4" event={"ID":"de2f5dab-53d4-4e0d-be0e-e71aa193705a","Type":"ContainerStarted","Data":"2ea3b1f97ced607252e4964dab4be31136e646690ec95d4086ed2bc478ef3234"} Nov 03 21:14:03.580707 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:03.578213 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ae29793-n5kjl" event={"ID":"07f1475c-e840-466e-bc78-75665a1468ba","Type":"ContainerStarted","Data":"43aeb73d697f519bcf8f5ef814cc90c590b5ce51030de9477c100a4d948c78a4"} Nov 03 21:14:04.586155 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:04.586056 4305 generic.go:356] "Generic (PLEG): container finished" podID="07f1475c-e840-466e-bc78-75665a1468ba" containerID="ce3da5d587a8ad4a8e661b41a01eb92efbec6492373146c315a943890f8bc78d" exitCode=0 Nov 03 21:14:04.586155 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:04.586135 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ae29793-n5kjl" event={"ID":"07f1475c-e840-466e-bc78-75665a1468ba","Type":"ContainerDied","Data":"ce3da5d587a8ad4a8e661b41a01eb92efbec6492373146c315a943890f8bc78d"} Nov 03 21:14:04.588060 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:04.588038 4305 generic.go:356] "Generic (PLEG): container finished" podID="de2f5dab-53d4-4e0d-be0e-e71aa193705a" containerID="658b0fafc8b622185d9dd861446d4b1e2b159b351dc530867ddd934c3daf88be" exitCode=0 Nov 03 21:14:04.588201 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:04.588127 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a7b6c5-4rgn4" event={"ID":"de2f5dab-53d4-4e0d-be0e-e71aa193705a","Type":"ContainerDied","Data":"658b0fafc8b622185d9dd861446d4b1e2b159b351dc530867ddd934c3daf88be"} Nov 03 21:14:04.912695 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:04.912628 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-e22abf-c94f8f5d8-n4hfh" podUID="6b5a3cd9-ead1-4bf0-ae49-f657449b964f" containerName="payload-processing" containerID="cri-o://0a3a97cbc251b50547be8030373868870c1595488e0f14eeaab882fc5c1c9bc9" gracePeriod=25 Nov 03 21:14:04.924351 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:04.924247 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dbbf182-5d1d-4a54-8f71-bb58afa654e1" path="/var/lib/kubelet/pods/4dbbf182-5d1d-4a54-8f71-bb58afa654e1/volumes" Nov 03 21:14:04.925044 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:04.925018 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c9106f1-54e4-4c54-bba9-938e47117a27" path="/var/lib/kubelet/pods/5c9106f1-54e4-4c54-bba9-938e47117a27/volumes" Nov 03 21:14:05.147713 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:05.147679 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:14:05.185876 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:05.185850 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-e22abf-c94f8f5d8-n4hfh" Nov 03 21:14:05.266277 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:05.266239 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b5a3cd9-ead1-4bf0-ae49-f657449b964f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b5a3cd9-ead1-4bf0-ae49-f657449b964f-plugins-config-volume\") pod \"6b5a3cd9-ead1-4bf0-ae49-f657449b964f\" (UID: \"6b5a3cd9-ead1-4bf0-ae49-f657449b964f\") " Nov 03 21:14:05.266475 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:05.266337 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b5a3cd9-ead1-4bf0-ae49-f657449b964f-kube-api-access-m7cns\" (UniqueName: \"kubernetes.io/projected/6b5a3cd9-ead1-4bf0-ae49-f657449b964f-kube-api-access-m7cns\") pod \"6b5a3cd9-ead1-4bf0-ae49-f657449b964f\" (UID: \"6b5a3cd9-ead1-4bf0-ae49-f657449b964f\") " Nov 03 21:14:05.266758 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:05.266732 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b5a3cd9-ead1-4bf0-ae49-f657449b964f-plugins-config-volume" pod "6b5a3cd9-ead1-4bf0-ae49-f657449b964f" (UID: "6b5a3cd9-ead1-4bf0-ae49-f657449b964f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:14:05.268896 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:05.268871 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b5a3cd9-ead1-4bf0-ae49-f657449b964f-kube-api-access-m7cns" pod "6b5a3cd9-ead1-4bf0-ae49-f657449b964f" (UID: "6b5a3cd9-ead1-4bf0-ae49-f657449b964f"). InnerVolumeSpecName "kube-api-access-m7cns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:14:05.367856 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:05.367825 4305 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b5a3cd9-ead1-4bf0-ae49-f657449b964f-plugins-config-volume\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:05.367856 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:05.367854 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7cns\" (UniqueName: \"kubernetes.io/projected/6b5a3cd9-ead1-4bf0-ae49-f657449b964f-kube-api-access-m7cns\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:05.601340 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:05.599865 4305 generic.go:356] "Generic (PLEG): container finished" podID="6b5a3cd9-ead1-4bf0-ae49-f657449b964f" containerID="0a3a97cbc251b50547be8030373868870c1595488e0f14eeaab882fc5c1c9bc9" exitCode=0 Nov 03 21:14:05.601340 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:05.600314 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-e22abf-c94f8f5d8-n4hfh" Nov 03 21:14:05.603697 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:05.602762 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-e22abf-c94f8f5d8-n4hfh" event={"ID":"6b5a3cd9-ead1-4bf0-ae49-f657449b964f","Type":"ContainerDied","Data":"0a3a97cbc251b50547be8030373868870c1595488e0f14eeaab882fc5c1c9bc9"} Nov 03 21:14:05.603697 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:05.602819 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-e22abf-c94f8f5d8-n4hfh" event={"ID":"6b5a3cd9-ead1-4bf0-ae49-f657449b964f","Type":"ContainerDied","Data":"958081c54739e39b56a553576a034a0d370a4e3543a5a596ff42a147bfb7969f"} Nov 03 21:14:05.603697 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:05.602848 4305 scope.go:122] "RemoveContainer" containerID="0a3a97cbc251b50547be8030373868870c1595488e0f14eeaab882fc5c1c9bc9" Nov 03 21:14:05.627749 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:05.627683 4305 scope.go:122] "RemoveContainer" containerID="0a3a97cbc251b50547be8030373868870c1595488e0f14eeaab882fc5c1c9bc9" Nov 03 21:14:05.629477 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:14:05.628142 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a3a97cbc251b50547be8030373868870c1595488e0f14eeaab882fc5c1c9bc9\": container with ID starting with 0a3a97cbc251b50547be8030373868870c1595488e0f14eeaab882fc5c1c9bc9 not found: ID does not exist" containerID="0a3a97cbc251b50547be8030373868870c1595488e0f14eeaab882fc5c1c9bc9" Nov 03 21:14:05.629477 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:05.628185 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a3a97cbc251b50547be8030373868870c1595488e0f14eeaab882fc5c1c9bc9"} err="failed to get container status \"0a3a97cbc251b50547be8030373868870c1595488e0f14eeaab882fc5c1c9bc9\": rpc error: code = NotFound desc = could not find container \"0a3a97cbc251b50547be8030373868870c1595488e0f14eeaab882fc5c1c9bc9\": container with ID starting with 0a3a97cbc251b50547be8030373868870c1595488e0f14eeaab882fc5c1c9bc9 not found: ID does not exist" Nov 03 21:14:05.632209 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:05.630702 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v"] Nov 03 21:14:05.632209 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:05.630974 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v" podUID="239c6c9d-3ac5-49b5-9093-e686f002ce2a" containerName="istio-proxy" containerID="cri-o://12c5a83a46c15be46fbf63f95d841479f29d7292d53fb8370968442f3eb2ea31" gracePeriod=30 Nov 03 21:14:05.777497 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:05.777210 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-e22abf-c94f8f5d8-n4hfh"] Nov 03 21:14:05.782676 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:05.782217 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-e22abf-c94f8f5d8-n4hfh"] Nov 03 21:14:05.817659 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:05.817629 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ae29793-n5kjl" Nov 03 21:14:05.865802 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:05.865732 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a7b6c5-4rgn4" Nov 03 21:14:05.982660 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:05.982622 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07f1475c-e840-466e-bc78-75665a1468ba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07f1475c-e840-466e-bc78-75665a1468ba-maas-api-openshift-ca\") pod \"07f1475c-e840-466e-bc78-75665a1468ba\" (UID: \"07f1475c-e840-466e-bc78-75665a1468ba\") " Nov 03 21:14:05.982821 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:05.982701 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07f1475c-e840-466e-bc78-75665a1468ba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07f1475c-e840-466e-bc78-75665a1468ba-maas-api-xks-ca\") pod \"07f1475c-e840-466e-bc78-75665a1468ba\" (UID: \"07f1475c-e840-466e-bc78-75665a1468ba\") " Nov 03 21:14:05.982888 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:05.982815 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de2f5dab-53d4-4e0d-be0e-e71aa193705a-kube-api-access-mh6jc\" (UniqueName: \"kubernetes.io/projected/de2f5dab-53d4-4e0d-be0e-e71aa193705a-kube-api-access-mh6jc\") pod \"de2f5dab-53d4-4e0d-be0e-e71aa193705a\" (UID: \"de2f5dab-53d4-4e0d-be0e-e71aa193705a\") " Nov 03 21:14:05.982925 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:05.982883 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07f1475c-e840-466e-bc78-75665a1468ba-kube-api-access-b44q5\" (UniqueName: \"kubernetes.io/projected/07f1475c-e840-466e-bc78-75665a1468ba-kube-api-access-b44q5\") pod \"07f1475c-e840-466e-bc78-75665a1468ba\" (UID: \"07f1475c-e840-466e-bc78-75665a1468ba\") " Nov 03 21:14:05.982967 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:05.982945 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de2f5dab-53d4-4e0d-be0e-e71aa193705a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de2f5dab-53d4-4e0d-be0e-e71aa193705a-maas-api-xks-ca\") pod \"de2f5dab-53d4-4e0d-be0e-e71aa193705a\" (UID: \"de2f5dab-53d4-4e0d-be0e-e71aa193705a\") " Nov 03 21:14:05.983012 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:05.983003 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de2f5dab-53d4-4e0d-be0e-e71aa193705a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de2f5dab-53d4-4e0d-be0e-e71aa193705a-maas-api-openshift-ca\") pod \"de2f5dab-53d4-4e0d-be0e-e71aa193705a\" (UID: \"de2f5dab-53d4-4e0d-be0e-e71aa193705a\") " Nov 03 21:14:05.983124 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:05.983064 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07f1475c-e840-466e-bc78-75665a1468ba-maas-api-openshift-ca" pod "07f1475c-e840-466e-bc78-75665a1468ba" (UID: "07f1475c-e840-466e-bc78-75665a1468ba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:14:05.983955 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:05.983421 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de2f5dab-53d4-4e0d-be0e-e71aa193705a-maas-api-openshift-ca" pod "de2f5dab-53d4-4e0d-be0e-e71aa193705a" (UID: "de2f5dab-53d4-4e0d-be0e-e71aa193705a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:14:05.983955 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:05.983709 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de2f5dab-53d4-4e0d-be0e-e71aa193705a-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:05.983955 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:05.983731 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07f1475c-e840-466e-bc78-75665a1468ba-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:05.985170 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:05.985143 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07f1475c-e840-466e-bc78-75665a1468ba-maas-api-xks-ca" pod "07f1475c-e840-466e-bc78-75665a1468ba" (UID: "07f1475c-e840-466e-bc78-75665a1468ba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:14:05.985626 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:05.985604 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07f1475c-e840-466e-bc78-75665a1468ba-kube-api-access-b44q5" pod "07f1475c-e840-466e-bc78-75665a1468ba" (UID: "07f1475c-e840-466e-bc78-75665a1468ba"). InnerVolumeSpecName "kube-api-access-b44q5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:14:05.985725 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:05.985702 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de2f5dab-53d4-4e0d-be0e-e71aa193705a-maas-api-xks-ca" pod "de2f5dab-53d4-4e0d-be0e-e71aa193705a" (UID: "de2f5dab-53d4-4e0d-be0e-e71aa193705a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:14:05.985796 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:05.985725 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de2f5dab-53d4-4e0d-be0e-e71aa193705a-kube-api-access-mh6jc" pod "de2f5dab-53d4-4e0d-be0e-e71aa193705a" (UID: "de2f5dab-53d4-4e0d-be0e-e71aa193705a"). InnerVolumeSpecName "kube-api-access-mh6jc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:14:06.085366 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:06.085333 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07f1475c-e840-466e-bc78-75665a1468ba-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:06.085366 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:06.085361 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mh6jc\" (UniqueName: \"kubernetes.io/projected/de2f5dab-53d4-4e0d-be0e-e71aa193705a-kube-api-access-mh6jc\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:06.085366 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:06.085374 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b44q5\" (UniqueName: \"kubernetes.io/projected/07f1475c-e840-466e-bc78-75665a1468ba-kube-api-access-b44q5\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:06.085658 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:06.085382 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de2f5dab-53d4-4e0d-be0e-e71aa193705a-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:06.609015 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:06.608925 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a7b6c5-4rgn4" Nov 03 21:14:06.609015 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:06.608944 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a7b6c5-4rgn4" event={"ID":"de2f5dab-53d4-4e0d-be0e-e71aa193705a","Type":"ContainerDied","Data":"2ea3b1f97ced607252e4964dab4be31136e646690ec95d4086ed2bc478ef3234"} Nov 03 21:14:06.609015 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:06.608993 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ea3b1f97ced607252e4964dab4be31136e646690ec95d4086ed2bc478ef3234" Nov 03 21:14:06.610474 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:06.610454 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ae29793-n5kjl" Nov 03 21:14:06.610474 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:06.610464 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ae29793-n5kjl" event={"ID":"07f1475c-e840-466e-bc78-75665a1468ba","Type":"ContainerDied","Data":"43aeb73d697f519bcf8f5ef814cc90c590b5ce51030de9477c100a4d948c78a4"} Nov 03 21:14:06.610676 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:06.610502 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43aeb73d697f519bcf8f5ef814cc90c590b5ce51030de9477c100a4d948c78a4" Nov 03 21:14:06.646947 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:06.646914 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-862138-k4bq6"] Nov 03 21:14:06.647473 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:06.647460 4305 cpu_manager.go:414] "containerMap: removing container" podUID="de2f5dab-53d4-4e0d-be0e-e71aa193705a" containerName="revoke-keys" Nov 03 21:14:06.647514 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:06.647476 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de2f5dab-53d4-4e0d-be0e-e71aa193705a" containerName="revoke-keys" Nov 03 21:14:06.647514 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:06.647490 4305 cpu_manager.go:414] "containerMap: removing container" podUID="07f1475c-e840-466e-bc78-75665a1468ba" containerName="revoke-keys" Nov 03 21:14:06.647514 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:06.647495 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07f1475c-e840-466e-bc78-75665a1468ba" containerName="revoke-keys" Nov 03 21:14:06.647514 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:06.647502 4305 cpu_manager.go:414] "containerMap: removing container" podUID="6b5a3cd9-ead1-4bf0-ae49-f657449b964f" containerName="payload-processing" Nov 03 21:14:06.647514 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:06.647507 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b5a3cd9-ead1-4bf0-ae49-f657449b964f" containerName="payload-processing" Nov 03 21:14:06.647679 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:06.647584 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="de2f5dab-53d4-4e0d-be0e-e71aa193705a" containerName="revoke-keys" Nov 03 21:14:06.647679 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:06.647592 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="07f1475c-e840-466e-bc78-75665a1468ba" containerName="revoke-keys" Nov 03 21:14:06.647679 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:06.647599 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b5a3cd9-ead1-4bf0-ae49-f657449b964f" containerName="payload-processing" Nov 03 21:14:06.651192 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:06.651167 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-862138-k4bq6" Nov 03 21:14:06.705186 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:06.704778 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:14:06.705186 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:06.704889 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:14:06.705186 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:06.704936 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77vtd\"" Nov 03 21:14:06.705186 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:06.705025 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:14:06.709280 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:06.709256 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-862138-k4bq6"] Nov 03 21:14:06.793677 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:06.793634 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a568044d-1a39-4916-86ea-fadfbcb6ddcd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-862138-k4bq6\" (UID: \"a568044d-1a39-4916-86ea-fadfbcb6ddcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-862138-k4bq6" Nov 03 21:14:06.793848 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:06.793829 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a568044d-1a39-4916-86ea-fadfbcb6ddcd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-862138-k4bq6\" (UID: \"a568044d-1a39-4916-86ea-fadfbcb6ddcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-862138-k4bq6" Nov 03 21:14:06.793950 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:06.793938 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp86h\" (UniqueName: \"kubernetes.io/projected/a568044d-1a39-4916-86ea-fadfbcb6ddcd-kube-api-access-mp86h\") pod \"maas-api-revoke-keys-e2e-worker-w2-862138-k4bq6\" (UID: \"a568044d-1a39-4916-86ea-fadfbcb6ddcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-862138-k4bq6" Nov 03 21:14:06.895196 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:06.895159 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a568044d-1a39-4916-86ea-fadfbcb6ddcd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-862138-k4bq6\" (UID: \"a568044d-1a39-4916-86ea-fadfbcb6ddcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-862138-k4bq6" Nov 03 21:14:06.895398 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:06.895264 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp86h\" (UniqueName: \"kubernetes.io/projected/a568044d-1a39-4916-86ea-fadfbcb6ddcd-kube-api-access-mp86h\") pod \"maas-api-revoke-keys-e2e-worker-w2-862138-k4bq6\" (UID: \"a568044d-1a39-4916-86ea-fadfbcb6ddcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-862138-k4bq6" Nov 03 21:14:06.895398 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:06.895337 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a568044d-1a39-4916-86ea-fadfbcb6ddcd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-862138-k4bq6\" (UID: \"a568044d-1a39-4916-86ea-fadfbcb6ddcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-862138-k4bq6" Nov 03 21:14:06.895683 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:06.895654 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b5a3cd9-ead1-4bf0-ae49-f657449b964f" path="/var/lib/kubelet/pods/6b5a3cd9-ead1-4bf0-ae49-f657449b964f/volumes" Nov 03 21:14:06.895890 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:06.895870 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a568044d-1a39-4916-86ea-fadfbcb6ddcd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-862138-k4bq6\" (UID: \"a568044d-1a39-4916-86ea-fadfbcb6ddcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-862138-k4bq6" Nov 03 21:14:06.898132 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:06.898107 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a568044d-1a39-4916-86ea-fadfbcb6ddcd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-862138-k4bq6\" (UID: \"a568044d-1a39-4916-86ea-fadfbcb6ddcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-862138-k4bq6" Nov 03 21:14:06.956440 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:06.956405 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp86h\" (UniqueName: \"kubernetes.io/projected/a568044d-1a39-4916-86ea-fadfbcb6ddcd-kube-api-access-mp86h\") pod \"maas-api-revoke-keys-e2e-worker-w2-862138-k4bq6\" (UID: \"a568044d-1a39-4916-86ea-fadfbcb6ddcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-862138-k4bq6" Nov 03 21:14:06.966142 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:06.966104 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-862138-k4bq6" Nov 03 21:14:07.348116 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:07.348087 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-862138-k4bq6"] Nov 03 21:14:07.349457 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:14:07.349416 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda568044d_1a39_4916_86ea_fadfbcb6ddcd.slice/crio-336091c6669e6c85c83eb7f1e66424b3425372f4c1ecc9eafb26341920746b7f WatchSource:0}: Error finding container 336091c6669e6c85c83eb7f1e66424b3425372f4c1ecc9eafb26341920746b7f: Status 404 returned error can't find the container with id 336091c6669e6c85c83eb7f1e66424b3425372f4c1ecc9eafb26341920746b7f Nov 03 21:14:07.618034 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:07.617942 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-862138-k4bq6" event={"ID":"a568044d-1a39-4916-86ea-fadfbcb6ddcd","Type":"ContainerStarted","Data":"336091c6669e6c85c83eb7f1e66424b3425372f4c1ecc9eafb26341920746b7f"} Nov 03 21:14:07.746973 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:07.746927 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" podUID="63a704b4-51ce-46c3-8007-34b308aff332" containerName="main" containerID="cri-o://cc6f011d0e33a6a978e37978709653548475ee42d7d839f30aa10f110a7a4a46" gracePeriod=45 Nov 03 21:14:07.765346 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:07.765216 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-12a423-559b96f96d-nmf2b"] Nov 03 21:14:07.778887 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:07.778847 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" podUID="7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3" containerName="main" containerID="cri-o://5722e751ffd83f95b108c793bf4e1583eeecd76bb01a8eb69c30a9d91c8d99b5" gracePeriod=45 Nov 03 21:14:07.822855 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:07.822815 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" podUID="63a704b4-51ce-46c3-8007-34b308aff332" containerName="main" probeResult="failure" output="Get \"https://10.134.0.101:8000/ready\": dial tcp 10.134.0.101:8000: connect: connection refused" Nov 03 21:14:07.878571 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:07.878490 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-12a423-846d4b6867-grwcg"] Nov 03 21:14:07.931863 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:07.931809 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" podUID="7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": dial tcp 10.134.0.61:8000: connect: connection refused" Nov 03 21:14:08.001940 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.001907 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93d44e89-kp8g9"] Nov 03 21:14:08.007001 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.006975 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d44e89-kp8g9" Nov 03 21:14:08.009861 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.008795 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6tbl\" (UniqueName: \"kubernetes.io/projected/da5ca27e-9637-4eea-b462-06f298f367de-kube-api-access-j6tbl\") pod \"maas-api-revoke-sub-93d44e89-kp8g9\" (UID: \"da5ca27e-9637-4eea-b462-06f298f367de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d44e89-kp8g9" Nov 03 21:14:08.009861 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.008871 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da5ca27e-9637-4eea-b462-06f298f367de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93d44e89-kp8g9\" (UID: \"da5ca27e-9637-4eea-b462-06f298f367de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d44e89-kp8g9" Nov 03 21:14:08.009861 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.008977 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da5ca27e-9637-4eea-b462-06f298f367de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93d44e89-kp8g9\" (UID: \"da5ca27e-9637-4eea-b462-06f298f367de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d44e89-kp8g9" Nov 03 21:14:08.054161 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.054122 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" Nov 03 21:14:08.067165 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.067137 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93d44e89-kp8g9"] Nov 03 21:14:08.072491 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.072467 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" Nov 03 21:14:08.109773 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.109741 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63a704b4-51ce-46c3-8007-34b308aff332-kube-api-access-x9x5q\" (UniqueName: \"kubernetes.io/projected/63a704b4-51ce-46c3-8007-34b308aff332-kube-api-access-x9x5q\") pod \"63a704b4-51ce-46c3-8007-34b308aff332\" (UID: \"63a704b4-51ce-46c3-8007-34b308aff332\") " Nov 03 21:14:08.109965 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.109792 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63a704b4-51ce-46c3-8007-34b308aff332-tls-certs\" (UniqueName: \"kubernetes.io/secret/63a704b4-51ce-46c3-8007-34b308aff332-tls-certs\") pod \"63a704b4-51ce-46c3-8007-34b308aff332\" (UID: \"63a704b4-51ce-46c3-8007-34b308aff332\") " Nov 03 21:14:08.109965 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.109842 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63a704b4-51ce-46c3-8007-34b308aff332-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63a704b4-51ce-46c3-8007-34b308aff332-model-cache\") pod \"63a704b4-51ce-46c3-8007-34b308aff332\" (UID: \"63a704b4-51ce-46c3-8007-34b308aff332\") " Nov 03 21:14:08.109965 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.109888 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63a704b4-51ce-46c3-8007-34b308aff332-dshm\" (UniqueName: \"kubernetes.io/empty-dir/63a704b4-51ce-46c3-8007-34b308aff332-dshm\") pod \"63a704b4-51ce-46c3-8007-34b308aff332\" (UID: \"63a704b4-51ce-46c3-8007-34b308aff332\") " Nov 03 21:14:08.109965 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.109917 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3-tmp-dir\") pod \"7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3\" (UID: \"7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3\") " Nov 03 21:14:08.109965 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.109957 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3-home\" (UniqueName: \"kubernetes.io/empty-dir/7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3-home\") pod \"7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3\" (UID: \"7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3\") " Nov 03 21:14:08.110251 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.110010 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3-model-cache\") pod \"7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3\" (UID: \"7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3\") " Nov 03 21:14:08.110251 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.110072 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3-tls-certs\" (UniqueName: \"kubernetes.io/secret/7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3-tls-certs\") pod \"7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3\" (UID: \"7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3\") " Nov 03 21:14:08.110251 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.110102 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63a704b4-51ce-46c3-8007-34b308aff332-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63a704b4-51ce-46c3-8007-34b308aff332-tmp-dir\") pod \"63a704b4-51ce-46c3-8007-34b308aff332\" (UID: \"63a704b4-51ce-46c3-8007-34b308aff332\") " Nov 03 21:14:08.110251 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.110165 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3-dshm\") pod \"7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3\" (UID: \"7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3\") " Nov 03 21:14:08.110251 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.110211 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3-kube-api-access-pxpd4\" (UniqueName: \"kubernetes.io/projected/7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3-kube-api-access-pxpd4\") pod \"7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3\" (UID: \"7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3\") " Nov 03 21:14:08.110530 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.110272 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63a704b4-51ce-46c3-8007-34b308aff332-home\" (UniqueName: \"kubernetes.io/empty-dir/63a704b4-51ce-46c3-8007-34b308aff332-home\") pod \"63a704b4-51ce-46c3-8007-34b308aff332\" (UID: \"63a704b4-51ce-46c3-8007-34b308aff332\") " Nov 03 21:14:08.110530 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.110155 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63a704b4-51ce-46c3-8007-34b308aff332-model-cache" pod "63a704b4-51ce-46c3-8007-34b308aff332" (UID: "63a704b4-51ce-46c3-8007-34b308aff332"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:08.110530 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.110377 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3-tmp-dir" pod "7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3" (UID: "7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:08.110530 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.110467 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da5ca27e-9637-4eea-b462-06f298f367de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93d44e89-kp8g9\" (UID: \"da5ca27e-9637-4eea-b462-06f298f367de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d44e89-kp8g9" Nov 03 21:14:08.110530 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.110485 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63a704b4-51ce-46c3-8007-34b308aff332-tmp-dir" pod "63a704b4-51ce-46c3-8007-34b308aff332" (UID: "63a704b4-51ce-46c3-8007-34b308aff332"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:08.110823 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.110632 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3-home" pod "7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3" (UID: "7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:08.110823 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.110647 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3-model-cache" pod "7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3" (UID: "7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:08.111026 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.111006 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6tbl\" (UniqueName: \"kubernetes.io/projected/da5ca27e-9637-4eea-b462-06f298f367de-kube-api-access-j6tbl\") pod \"maas-api-revoke-sub-93d44e89-kp8g9\" (UID: \"da5ca27e-9637-4eea-b462-06f298f367de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d44e89-kp8g9" Nov 03 21:14:08.111100 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.111086 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da5ca27e-9637-4eea-b462-06f298f367de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93d44e89-kp8g9\" (UID: \"da5ca27e-9637-4eea-b462-06f298f367de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d44e89-kp8g9" Nov 03 21:14:08.112978 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.111292 4305 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63a704b4-51ce-46c3-8007-34b308aff332-model-cache\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:08.112978 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.111315 4305 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3-tmp-dir\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:08.112978 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.111330 4305 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3-home\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:08.112978 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.111344 4305 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3-model-cache\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:08.112978 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.111361 4305 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63a704b4-51ce-46c3-8007-34b308aff332-tmp-dir\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:08.112978 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.112290 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da5ca27e-9637-4eea-b462-06f298f367de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93d44e89-kp8g9\" (UID: \"da5ca27e-9637-4eea-b462-06f298f367de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d44e89-kp8g9" Nov 03 21:14:08.112978 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.112938 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63a704b4-51ce-46c3-8007-34b308aff332-home" pod "63a704b4-51ce-46c3-8007-34b308aff332" (UID: "63a704b4-51ce-46c3-8007-34b308aff332"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:08.113419 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.113108 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63a704b4-51ce-46c3-8007-34b308aff332-tls-certs" pod "63a704b4-51ce-46c3-8007-34b308aff332" (UID: "63a704b4-51ce-46c3-8007-34b308aff332"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:14:08.113419 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.113116 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63a704b4-51ce-46c3-8007-34b308aff332-kube-api-access-x9x5q" pod "63a704b4-51ce-46c3-8007-34b308aff332" (UID: "63a704b4-51ce-46c3-8007-34b308aff332"). InnerVolumeSpecName "kube-api-access-x9x5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:14:08.113419 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.113327 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3-tls-certs" pod "7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3" (UID: "7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:14:08.113702 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.113679 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da5ca27e-9637-4eea-b462-06f298f367de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93d44e89-kp8g9\" (UID: \"da5ca27e-9637-4eea-b462-06f298f367de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d44e89-kp8g9" Nov 03 21:14:08.114062 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.114036 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3-dshm" pod "7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3" (UID: "7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:08.115135 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.115115 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63a704b4-51ce-46c3-8007-34b308aff332-dshm" pod "63a704b4-51ce-46c3-8007-34b308aff332" (UID: "63a704b4-51ce-46c3-8007-34b308aff332"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:08.115313 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.115299 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3-kube-api-access-pxpd4" pod "7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3" (UID: "7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3"). InnerVolumeSpecName "kube-api-access-pxpd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:14:08.188896 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.188862 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6tbl\" (UniqueName: \"kubernetes.io/projected/da5ca27e-9637-4eea-b462-06f298f367de-kube-api-access-j6tbl\") pod \"maas-api-revoke-sub-93d44e89-kp8g9\" (UID: \"da5ca27e-9637-4eea-b462-06f298f367de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d44e89-kp8g9" Nov 03 21:14:08.213152 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.213100 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9x5q\" (UniqueName: \"kubernetes.io/projected/63a704b4-51ce-46c3-8007-34b308aff332-kube-api-access-x9x5q\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:08.213152 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.213147 4305 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63a704b4-51ce-46c3-8007-34b308aff332-tls-certs\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:08.213487 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.213162 4305 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63a704b4-51ce-46c3-8007-34b308aff332-dshm\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:08.213487 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.213176 4305 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3-tls-certs\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:08.213487 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.213191 4305 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3-dshm\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:08.213487 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.213205 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxpd4\" (UniqueName: \"kubernetes.io/projected/7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3-kube-api-access-pxpd4\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:08.213487 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.213217 4305 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63a704b4-51ce-46c3-8007-34b308aff332-home\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:08.352601 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.352559 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d44e89-kp8g9" Nov 03 21:14:08.627660 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.627578 4305 generic.go:356] "Generic (PLEG): container finished" podID="a568044d-1a39-4916-86ea-fadfbcb6ddcd" containerID="59e7ac1ddd423b18db71868de5bc2d66bf460e7e17d060598d6b128b15e8d0d0" exitCode=0 Nov 03 21:14:08.628112 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.627661 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-862138-k4bq6" event={"ID":"a568044d-1a39-4916-86ea-fadfbcb6ddcd","Type":"ContainerDied","Data":"59e7ac1ddd423b18db71868de5bc2d66bf460e7e17d060598d6b128b15e8d0d0"} Nov 03 21:14:08.629061 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.629040 4305 generic.go:356] "Generic (PLEG): container finished" podID="63a704b4-51ce-46c3-8007-34b308aff332" containerID="cc6f011d0e33a6a978e37978709653548475ee42d7d839f30aa10f110a7a4a46" exitCode=0 Nov 03 21:14:08.629194 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.629082 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" Nov 03 21:14:08.629194 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.629117 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" event={"ID":"63a704b4-51ce-46c3-8007-34b308aff332","Type":"ContainerDied","Data":"cc6f011d0e33a6a978e37978709653548475ee42d7d839f30aa10f110a7a4a46"} Nov 03 21:14:08.629194 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.629158 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8" event={"ID":"63a704b4-51ce-46c3-8007-34b308aff332","Type":"ContainerDied","Data":"d3778305d329f7a06c9bec3b2b8eb69e06bfd6a597d952f3f096af0d78d107ff"} Nov 03 21:14:08.629194 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.629184 4305 scope.go:122] "RemoveContainer" containerID="cc6f011d0e33a6a978e37978709653548475ee42d7d839f30aa10f110a7a4a46" Nov 03 21:14:08.630588 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.630566 4305 generic.go:356] "Generic (PLEG): container finished" podID="7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3" containerID="5722e751ffd83f95b108c793bf4e1583eeecd76bb01a8eb69c30a9d91c8d99b5" exitCode=0 Nov 03 21:14:08.630705 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.630595 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" event={"ID":"7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3","Type":"ContainerDied","Data":"5722e751ffd83f95b108c793bf4e1583eeecd76bb01a8eb69c30a9d91c8d99b5"} Nov 03 21:14:08.630705 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.630609 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" Nov 03 21:14:08.630705 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.630633 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq" event={"ID":"7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3","Type":"ContainerDied","Data":"f931558a9ab1e734a5fff2b9dad210cecf516a4ca147663515c05b9c43024670"} Nov 03 21:14:08.640761 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.640742 4305 scope.go:122] "RemoveContainer" containerID="cc6f011d0e33a6a978e37978709653548475ee42d7d839f30aa10f110a7a4a46" Nov 03 21:14:08.641088 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:14:08.641068 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc6f011d0e33a6a978e37978709653548475ee42d7d839f30aa10f110a7a4a46\": container with ID starting with cc6f011d0e33a6a978e37978709653548475ee42d7d839f30aa10f110a7a4a46 not found: ID does not exist" containerID="cc6f011d0e33a6a978e37978709653548475ee42d7d839f30aa10f110a7a4a46" Nov 03 21:14:08.641191 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.641099 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc6f011d0e33a6a978e37978709653548475ee42d7d839f30aa10f110a7a4a46"} err="failed to get container status \"cc6f011d0e33a6a978e37978709653548475ee42d7d839f30aa10f110a7a4a46\": rpc error: code = NotFound desc = could not find container \"cc6f011d0e33a6a978e37978709653548475ee42d7d839f30aa10f110a7a4a46\": container with ID starting with cc6f011d0e33a6a978e37978709653548475ee42d7d839f30aa10f110a7a4a46 not found: ID does not exist" Nov 03 21:14:08.641191 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.641118 4305 scope.go:122] "RemoveContainer" containerID="5722e751ffd83f95b108c793bf4e1583eeecd76bb01a8eb69c30a9d91c8d99b5" Nov 03 21:14:08.651274 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.651254 4305 scope.go:122] "RemoveContainer" containerID="5722e751ffd83f95b108c793bf4e1583eeecd76bb01a8eb69c30a9d91c8d99b5" Nov 03 21:14:08.651707 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:14:08.651685 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5722e751ffd83f95b108c793bf4e1583eeecd76bb01a8eb69c30a9d91c8d99b5\": container with ID starting with 5722e751ffd83f95b108c793bf4e1583eeecd76bb01a8eb69c30a9d91c8d99b5 not found: ID does not exist" containerID="5722e751ffd83f95b108c793bf4e1583eeecd76bb01a8eb69c30a9d91c8d99b5" Nov 03 21:14:08.651770 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.651717 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5722e751ffd83f95b108c793bf4e1583eeecd76bb01a8eb69c30a9d91c8d99b5"} err="failed to get container status \"5722e751ffd83f95b108c793bf4e1583eeecd76bb01a8eb69c30a9d91c8d99b5\": rpc error: code = NotFound desc = could not find container \"5722e751ffd83f95b108c793bf4e1583eeecd76bb01a8eb69c30a9d91c8d99b5\": container with ID starting with 5722e751ffd83f95b108c793bf4e1583eeecd76bb01a8eb69c30a9d91c8d99b5 not found: ID does not exist" Nov 03 21:14:08.744043 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.744017 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93d44e89-kp8g9"] Nov 03 21:14:08.745094 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:14:08.745070 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda5ca27e_9637_4eea_b462_06f298f367de.slice/crio-a9cdcd48edc37fbe6a9db77bb43dbd1ed207f07b0b2dfd021a174b98a9334d0c WatchSource:0}: Error finding container a9cdcd48edc37fbe6a9db77bb43dbd1ed207f07b0b2dfd021a174b98a9334d0c: Status 404 returned error can't find the container with id a9cdcd48edc37fbe6a9db77bb43dbd1ed207f07b0b2dfd021a174b98a9334d0c Nov 03 21:14:08.867354 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.867313 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq"] Nov 03 21:14:08.871899 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.871870 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-e22abf/premium-w5-e22abf-kserve-f487d7c4c-sz9xq"] Nov 03 21:14:08.896037 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.896006 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3" path="/var/lib/kubelet/pods/7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3/volumes" Nov 03 21:14:08.979746 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.979699 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8"] Nov 03 21:14:08.984496 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:08.984458 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-e22abf/distinct-2-w5-e22abf-kserve-6df8b49f69-zmsl8"] Nov 03 21:14:09.650708 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:09.650674 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d44e89-kp8g9" event={"ID":"da5ca27e-9637-4eea-b462-06f298f367de","Type":"ContainerStarted","Data":"4b6e94af048ab5358be8f1942e21d34c8104c475f0246e1367864ab94a7cc6e1"} Nov 03 21:14:09.651248 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:09.650749 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d44e89-kp8g9" event={"ID":"da5ca27e-9637-4eea-b462-06f298f367de","Type":"ContainerStarted","Data":"a9cdcd48edc37fbe6a9db77bb43dbd1ed207f07b0b2dfd021a174b98a9334d0c"} Nov 03 21:14:09.773044 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:09.772979 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d44e89-kp8g9" podStartSLOduration=2.7729596279999997 podStartE2EDuration="2.772959628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:14:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:14:09.770285433 +0000 UTC m=+2084.995605267" watchObservedRunningTime="2026-11-03 21:14:09.772959628 +0000 UTC m=+2084.998279420" Nov 03 21:14:09.851002 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:09.850978 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-862138-k4bq6" Nov 03 21:14:09.951931 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:09.951841 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a568044d-1a39-4916-86ea-fadfbcb6ddcd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a568044d-1a39-4916-86ea-fadfbcb6ddcd-maas-api-openshift-ca\") pod \"a568044d-1a39-4916-86ea-fadfbcb6ddcd\" (UID: \"a568044d-1a39-4916-86ea-fadfbcb6ddcd\") " Nov 03 21:14:09.951931 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:09.951912 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a568044d-1a39-4916-86ea-fadfbcb6ddcd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a568044d-1a39-4916-86ea-fadfbcb6ddcd-maas-api-xks-ca\") pod \"a568044d-1a39-4916-86ea-fadfbcb6ddcd\" (UID: \"a568044d-1a39-4916-86ea-fadfbcb6ddcd\") " Nov 03 21:14:09.952126 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:09.952004 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a568044d-1a39-4916-86ea-fadfbcb6ddcd-kube-api-access-mp86h\" (UniqueName: \"kubernetes.io/projected/a568044d-1a39-4916-86ea-fadfbcb6ddcd-kube-api-access-mp86h\") pod \"a568044d-1a39-4916-86ea-fadfbcb6ddcd\" (UID: \"a568044d-1a39-4916-86ea-fadfbcb6ddcd\") " Nov 03 21:14:09.952284 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:09.952257 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a568044d-1a39-4916-86ea-fadfbcb6ddcd-maas-api-openshift-ca" pod "a568044d-1a39-4916-86ea-fadfbcb6ddcd" (UID: "a568044d-1a39-4916-86ea-fadfbcb6ddcd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:14:09.952490 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:09.952429 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a568044d-1a39-4916-86ea-fadfbcb6ddcd-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:09.954136 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:09.954115 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a568044d-1a39-4916-86ea-fadfbcb6ddcd-maas-api-xks-ca" pod "a568044d-1a39-4916-86ea-fadfbcb6ddcd" (UID: "a568044d-1a39-4916-86ea-fadfbcb6ddcd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:14:09.954291 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:09.954269 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a568044d-1a39-4916-86ea-fadfbcb6ddcd-kube-api-access-mp86h" pod "a568044d-1a39-4916-86ea-fadfbcb6ddcd" (UID: "a568044d-1a39-4916-86ea-fadfbcb6ddcd"). InnerVolumeSpecName "kube-api-access-mp86h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:14:10.053486 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:10.053410 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp86h\" (UniqueName: \"kubernetes.io/projected/a568044d-1a39-4916-86ea-fadfbcb6ddcd-kube-api-access-mp86h\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:10.053486 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:10.053483 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a568044d-1a39-4916-86ea-fadfbcb6ddcd-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:10.111860 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:10.111824 4305 patch_prober.go:29] interesting pod/e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:14:10.112035 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:10.111888 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v" podUID="239c6c9d-3ac5-49b5-9093-e686f002ce2a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:14:10.658687 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:10.658655 4305 generic.go:356] "Generic (PLEG): container finished" podID="da5ca27e-9637-4eea-b462-06f298f367de" containerID="4b6e94af048ab5358be8f1942e21d34c8104c475f0246e1367864ab94a7cc6e1" exitCode=0 Nov 03 21:14:10.659103 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:10.658736 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d44e89-kp8g9" event={"ID":"da5ca27e-9637-4eea-b462-06f298f367de","Type":"ContainerDied","Data":"4b6e94af048ab5358be8f1942e21d34c8104c475f0246e1367864ab94a7cc6e1"} Nov 03 21:14:10.660156 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:10.660131 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-862138-k4bq6" event={"ID":"a568044d-1a39-4916-86ea-fadfbcb6ddcd","Type":"ContainerDied","Data":"336091c6669e6c85c83eb7f1e66424b3425372f4c1ecc9eafb26341920746b7f"} Nov 03 21:14:10.660283 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:10.660162 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-862138-k4bq6" Nov 03 21:14:10.660283 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:10.660178 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="336091c6669e6c85c83eb7f1e66424b3425372f4c1ecc9eafb26341920746b7f" Nov 03 21:14:10.889610 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:10.889585 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v" Nov 03 21:14:10.895480 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:10.895408 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63a704b4-51ce-46c3-8007-34b308aff332" path="/var/lib/kubelet/pods/63a704b4-51ce-46c3-8007-34b308aff332/volumes" Nov 03 21:14:10.963040 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:10.963004 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/239c6c9d-3ac5-49b5-9093-e686f002ce2a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/239c6c9d-3ac5-49b5-9093-e686f002ce2a-istio-envoy\") pod \"239c6c9d-3ac5-49b5-9093-e686f002ce2a\" (UID: \"239c6c9d-3ac5-49b5-9093-e686f002ce2a\") " Nov 03 21:14:10.963246 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:10.963083 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/239c6c9d-3ac5-49b5-9093-e686f002ce2a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/239c6c9d-3ac5-49b5-9093-e686f002ce2a-istiod-ca-cert\") pod \"239c6c9d-3ac5-49b5-9093-e686f002ce2a\" (UID: \"239c6c9d-3ac5-49b5-9093-e686f002ce2a\") " Nov 03 21:14:10.963246 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:10.963156 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/239c6c9d-3ac5-49b5-9093-e686f002ce2a-kube-api-access-5px9d\" (UniqueName: \"kubernetes.io/projected/239c6c9d-3ac5-49b5-9093-e686f002ce2a-kube-api-access-5px9d\") pod \"239c6c9d-3ac5-49b5-9093-e686f002ce2a\" (UID: \"239c6c9d-3ac5-49b5-9093-e686f002ce2a\") " Nov 03 21:14:10.963246 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:10.963189 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/239c6c9d-3ac5-49b5-9093-e686f002ce2a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/239c6c9d-3ac5-49b5-9093-e686f002ce2a-workload-certs\") pod \"239c6c9d-3ac5-49b5-9093-e686f002ce2a\" (UID: \"239c6c9d-3ac5-49b5-9093-e686f002ce2a\") " Nov 03 21:14:10.963246 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:10.963236 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/239c6c9d-3ac5-49b5-9093-e686f002ce2a-istio-token\" (UniqueName: \"kubernetes.io/projected/239c6c9d-3ac5-49b5-9093-e686f002ce2a-istio-token\") pod \"239c6c9d-3ac5-49b5-9093-e686f002ce2a\" (UID: \"239c6c9d-3ac5-49b5-9093-e686f002ce2a\") " Nov 03 21:14:10.963492 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:10.963318 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/239c6c9d-3ac5-49b5-9093-e686f002ce2a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/239c6c9d-3ac5-49b5-9093-e686f002ce2a-credential-socket\") pod \"239c6c9d-3ac5-49b5-9093-e686f002ce2a\" (UID: \"239c6c9d-3ac5-49b5-9093-e686f002ce2a\") " Nov 03 21:14:10.963492 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:10.963384 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/239c6c9d-3ac5-49b5-9093-e686f002ce2a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/239c6c9d-3ac5-49b5-9093-e686f002ce2a-istio-podinfo\") pod \"239c6c9d-3ac5-49b5-9093-e686f002ce2a\" (UID: \"239c6c9d-3ac5-49b5-9093-e686f002ce2a\") " Nov 03 21:14:10.963492 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:10.963415 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/239c6c9d-3ac5-49b5-9093-e686f002ce2a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/239c6c9d-3ac5-49b5-9093-e686f002ce2a-workload-socket\") pod \"239c6c9d-3ac5-49b5-9093-e686f002ce2a\" (UID: \"239c6c9d-3ac5-49b5-9093-e686f002ce2a\") " Nov 03 21:14:10.963641 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:10.963507 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/239c6c9d-3ac5-49b5-9093-e686f002ce2a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/239c6c9d-3ac5-49b5-9093-e686f002ce2a-istio-data\") pod \"239c6c9d-3ac5-49b5-9093-e686f002ce2a\" (UID: \"239c6c9d-3ac5-49b5-9093-e686f002ce2a\") " Nov 03 21:14:10.963641 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:10.963526 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/239c6c9d-3ac5-49b5-9093-e686f002ce2a-workload-certs" pod "239c6c9d-3ac5-49b5-9093-e686f002ce2a" (UID: "239c6c9d-3ac5-49b5-9093-e686f002ce2a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:10.963641 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:10.963551 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/239c6c9d-3ac5-49b5-9093-e686f002ce2a-istiod-ca-cert" pod "239c6c9d-3ac5-49b5-9093-e686f002ce2a" (UID: "239c6c9d-3ac5-49b5-9093-e686f002ce2a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:14:10.963791 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:10.963701 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/239c6c9d-3ac5-49b5-9093-e686f002ce2a-credential-socket" pod "239c6c9d-3ac5-49b5-9093-e686f002ce2a" (UID: "239c6c9d-3ac5-49b5-9093-e686f002ce2a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:10.963791 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:10.963760 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/239c6c9d-3ac5-49b5-9093-e686f002ce2a-workload-socket" pod "239c6c9d-3ac5-49b5-9093-e686f002ce2a" (UID: "239c6c9d-3ac5-49b5-9093-e686f002ce2a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:10.963911 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:10.963870 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/239c6c9d-3ac5-49b5-9093-e686f002ce2a-istio-data" pod "239c6c9d-3ac5-49b5-9093-e686f002ce2a" (UID: "239c6c9d-3ac5-49b5-9093-e686f002ce2a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:10.964235 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:10.964090 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/239c6c9d-3ac5-49b5-9093-e686f002ce2a-workload-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:10.964235 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:10.964147 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/239c6c9d-3ac5-49b5-9093-e686f002ce2a-istio-data\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:10.964235 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:10.964163 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/239c6c9d-3ac5-49b5-9093-e686f002ce2a-istiod-ca-cert\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:10.964235 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:10.964177 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/239c6c9d-3ac5-49b5-9093-e686f002ce2a-workload-certs\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:10.964235 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:10.964191 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/239c6c9d-3ac5-49b5-9093-e686f002ce2a-credential-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:10.965759 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:10.965733 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/239c6c9d-3ac5-49b5-9093-e686f002ce2a-kube-api-access-5px9d" pod "239c6c9d-3ac5-49b5-9093-e686f002ce2a" (UID: "239c6c9d-3ac5-49b5-9093-e686f002ce2a"). InnerVolumeSpecName "kube-api-access-5px9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:14:10.965841 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:10.965783 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/239c6c9d-3ac5-49b5-9093-e686f002ce2a-istio-envoy" pod "239c6c9d-3ac5-49b5-9093-e686f002ce2a" (UID: "239c6c9d-3ac5-49b5-9093-e686f002ce2a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:10.965841 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:10.965783 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/239c6c9d-3ac5-49b5-9093-e686f002ce2a-istio-podinfo" pod "239c6c9d-3ac5-49b5-9093-e686f002ce2a" (UID: "239c6c9d-3ac5-49b5-9093-e686f002ce2a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:14:10.966142 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:10.966127 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/239c6c9d-3ac5-49b5-9093-e686f002ce2a-istio-token" pod "239c6c9d-3ac5-49b5-9093-e686f002ce2a" (UID: "239c6c9d-3ac5-49b5-9093-e686f002ce2a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:14:11.065323 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:11.065284 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/239c6c9d-3ac5-49b5-9093-e686f002ce2a-istio-envoy\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:11.065323 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:11.065323 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5px9d\" (UniqueName: \"kubernetes.io/projected/239c6c9d-3ac5-49b5-9093-e686f002ce2a-kube-api-access-5px9d\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:11.065639 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:11.065339 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/239c6c9d-3ac5-49b5-9093-e686f002ce2a-istio-token\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:11.065639 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:11.065353 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/239c6c9d-3ac5-49b5-9093-e686f002ce2a-istio-podinfo\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:11.668900 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:11.668868 4305 generic.go:356] "Generic (PLEG): container finished" podID="239c6c9d-3ac5-49b5-9093-e686f002ce2a" containerID="12c5a83a46c15be46fbf63f95d841479f29d7292d53fb8370968442f3eb2ea31" exitCode=0 Nov 03 21:14:11.669378 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:11.668926 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v" Nov 03 21:14:11.669378 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:11.668949 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v" event={"ID":"239c6c9d-3ac5-49b5-9093-e686f002ce2a","Type":"ContainerDied","Data":"12c5a83a46c15be46fbf63f95d841479f29d7292d53fb8370968442f3eb2ea31"} Nov 03 21:14:11.669378 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:11.669004 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v" event={"ID":"239c6c9d-3ac5-49b5-9093-e686f002ce2a","Type":"ContainerDied","Data":"9d91e4c52012709bd105882b8a2cff3b0a018496cfac47e1f4d50298ed413ba2"} Nov 03 21:14:11.669378 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:11.669026 4305 scope.go:122] "RemoveContainer" containerID="12c5a83a46c15be46fbf63f95d841479f29d7292d53fb8370968442f3eb2ea31" Nov 03 21:14:11.681290 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:11.681267 4305 scope.go:122] "RemoveContainer" containerID="0eeba93b3f6bc986d8eda7e3d9427349d5e1c7704233e90443cdd1a54dd67d1d" Nov 03 21:14:11.691699 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:11.691674 4305 scope.go:122] "RemoveContainer" containerID="12c5a83a46c15be46fbf63f95d841479f29d7292d53fb8370968442f3eb2ea31" Nov 03 21:14:11.692081 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:14:11.692059 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12c5a83a46c15be46fbf63f95d841479f29d7292d53fb8370968442f3eb2ea31\": container with ID starting with 12c5a83a46c15be46fbf63f95d841479f29d7292d53fb8370968442f3eb2ea31 not found: ID does not exist" containerID="12c5a83a46c15be46fbf63f95d841479f29d7292d53fb8370968442f3eb2ea31" Nov 03 21:14:11.692171 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:11.692093 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12c5a83a46c15be46fbf63f95d841479f29d7292d53fb8370968442f3eb2ea31"} err="failed to get container status \"12c5a83a46c15be46fbf63f95d841479f29d7292d53fb8370968442f3eb2ea31\": rpc error: code = NotFound desc = could not find container \"12c5a83a46c15be46fbf63f95d841479f29d7292d53fb8370968442f3eb2ea31\": container with ID starting with 12c5a83a46c15be46fbf63f95d841479f29d7292d53fb8370968442f3eb2ea31 not found: ID does not exist" Nov 03 21:14:11.692171 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:11.692113 4305 scope.go:122] "RemoveContainer" containerID="0eeba93b3f6bc986d8eda7e3d9427349d5e1c7704233e90443cdd1a54dd67d1d" Nov 03 21:14:11.692563 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:14:11.692538 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0eeba93b3f6bc986d8eda7e3d9427349d5e1c7704233e90443cdd1a54dd67d1d\": container with ID starting with 0eeba93b3f6bc986d8eda7e3d9427349d5e1c7704233e90443cdd1a54dd67d1d not found: ID does not exist" containerID="0eeba93b3f6bc986d8eda7e3d9427349d5e1c7704233e90443cdd1a54dd67d1d" Nov 03 21:14:11.692637 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:11.692574 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eeba93b3f6bc986d8eda7e3d9427349d5e1c7704233e90443cdd1a54dd67d1d"} err="failed to get container status \"0eeba93b3f6bc986d8eda7e3d9427349d5e1c7704233e90443cdd1a54dd67d1d\": rpc error: code = NotFound desc = could not find container \"0eeba93b3f6bc986d8eda7e3d9427349d5e1c7704233e90443cdd1a54dd67d1d\": container with ID starting with 0eeba93b3f6bc986d8eda7e3d9427349d5e1c7704233e90443cdd1a54dd67d1d not found: ID does not exist" Nov 03 21:14:11.818210 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:11.818182 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d44e89-kp8g9" Nov 03 21:14:11.834838 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:11.834807 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v"] Nov 03 21:14:11.872579 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:11.872543 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da5ca27e-9637-4eea-b462-06f298f367de-kube-api-access-j6tbl\" (UniqueName: \"kubernetes.io/projected/da5ca27e-9637-4eea-b462-06f298f367de-kube-api-access-j6tbl\") pod \"da5ca27e-9637-4eea-b462-06f298f367de\" (UID: \"da5ca27e-9637-4eea-b462-06f298f367de\") " Nov 03 21:14:11.872796 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:11.872726 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da5ca27e-9637-4eea-b462-06f298f367de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da5ca27e-9637-4eea-b462-06f298f367de-maas-api-xks-ca\") pod \"da5ca27e-9637-4eea-b462-06f298f367de\" (UID: \"da5ca27e-9637-4eea-b462-06f298f367de\") " Nov 03 21:14:11.872796 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:11.872781 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da5ca27e-9637-4eea-b462-06f298f367de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da5ca27e-9637-4eea-b462-06f298f367de-maas-api-openshift-ca\") pod \"da5ca27e-9637-4eea-b462-06f298f367de\" (UID: \"da5ca27e-9637-4eea-b462-06f298f367de\") " Nov 03 21:14:11.873152 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:11.873128 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da5ca27e-9637-4eea-b462-06f298f367de-maas-api-openshift-ca" pod "da5ca27e-9637-4eea-b462-06f298f367de" (UID: "da5ca27e-9637-4eea-b462-06f298f367de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:14:11.874876 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:11.874855 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da5ca27e-9637-4eea-b462-06f298f367de-maas-api-xks-ca" pod "da5ca27e-9637-4eea-b462-06f298f367de" (UID: "da5ca27e-9637-4eea-b462-06f298f367de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:14:11.874949 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:11.874924 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da5ca27e-9637-4eea-b462-06f298f367de-kube-api-access-j6tbl" pod "da5ca27e-9637-4eea-b462-06f298f367de" (UID: "da5ca27e-9637-4eea-b462-06f298f367de"). InnerVolumeSpecName "kube-api-access-j6tbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:14:11.884585 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:11.884558 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-01c7a3-openshift-default-65f658b97f-6w66v"] Nov 03 21:14:11.975015 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:11.974930 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da5ca27e-9637-4eea-b462-06f298f367de-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:11.975015 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:11.974964 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da5ca27e-9637-4eea-b462-06f298f367de-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:11.975015 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:11.974974 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6tbl\" (UniqueName: \"kubernetes.io/projected/da5ca27e-9637-4eea-b462-06f298f367de-kube-api-access-j6tbl\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:12.180120 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:12.180079 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-27ssq"] Nov 03 21:14:12.180341 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:12.180305 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-27ssq" podUID="8f183a1c-3354-4637-891c-9bcbfb0daf67" containerName="istio-proxy" containerID="cri-o://134d2d0e29ddca31f2a1783a064d951bb0fb94713b9c2e4ed036e16bda28bdb2" gracePeriod=30 Nov 03 21:14:12.185993 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:12.185960 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd"] Nov 03 21:14:12.186677 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:12.186646 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd" podUID="c97844f3-937c-4873-865a-c408d83b0890" containerName="istio-proxy" containerID="cri-o://3c92e1c0e4ef243987942b115cada19a70ea9c7b130d49dc0a0f47344c6987bd" gracePeriod=30 Nov 03 21:14:12.190962 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:12.190935 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-x954c"] Nov 03 21:14:12.191190 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:12.191158 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-x954c" podUID="2195a2fd-e42f-4e90-861d-a50b75f81973" containerName="istio-proxy" containerID="cri-o://38753558348ff64f5e879a4041a51532b05251bd1d16f6c0200cca6e992cd8ef" gracePeriod=30 Nov 03 21:14:12.195276 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:12.195151 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn"] Nov 03 21:14:12.196515 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:12.195370 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn" podUID="fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc" containerName="istio-proxy" containerID="cri-o://32dca95a4802050f05da86005a6ea74031dc7b0017bbe0276b03e076fefd6383" gracePeriod=30 Nov 03 21:14:12.677103 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:12.677068 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d44e89-kp8g9" Nov 03 21:14:12.677103 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:12.677091 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93d44e89-kp8g9" event={"ID":"da5ca27e-9637-4eea-b462-06f298f367de","Type":"ContainerDied","Data":"a9cdcd48edc37fbe6a9db77bb43dbd1ed207f07b0b2dfd021a174b98a9334d0c"} Nov 03 21:14:12.677659 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:12.677183 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9cdcd48edc37fbe6a9db77bb43dbd1ed207f07b0b2dfd021a174b98a9334d0c" Nov 03 21:14:12.749375 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:12.749334 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:14:12.749567 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:12.749398 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd" podUID="c97844f3-937c-4873-865a-c408d83b0890" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:14:12.765522 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:12.765485 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-12a423-559b96f96d-nmf2b" podUID="3e6b8d49-0db7-4bbc-9b3e-1105d2bdbc78" containerName="payload-processing" containerID="cri-o://3f479753f615643b9fe749dbdaeb5ddde4eace1984c7faab8edcedfef05c9ea3" gracePeriod=25 Nov 03 21:14:12.879378 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:12.879328 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-12a423-846d4b6867-grwcg" podUID="7b0135ee-b547-46e8-9953-6a46cb61f2c4" containerName="payload-pre-processing" containerID="cri-o://e69e1c1980274f4b77b456df4c88e10de37015cb0a6d99dfa49f4a8f433dfd4c" gracePeriod=25 Nov 03 21:14:12.895134 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:12.895099 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="239c6c9d-3ac5-49b5-9093-e686f002ce2a" path="/var/lib/kubelet/pods/239c6c9d-3ac5-49b5-9093-e686f002ce2a/volumes" Nov 03 21:14:16.322155 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:14:16.322121 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:14:17.497612 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.497577 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-27ssq" Nov 03 21:14:17.540793 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.540763 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f183a1c-3354-4637-891c-9bcbfb0daf67-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f183a1c-3354-4637-891c-9bcbfb0daf67-istio-envoy\") pod \"8f183a1c-3354-4637-891c-9bcbfb0daf67\" (UID: \"8f183a1c-3354-4637-891c-9bcbfb0daf67\") " Nov 03 21:14:17.540966 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.540886 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f183a1c-3354-4637-891c-9bcbfb0daf67-istio-token\" (UniqueName: \"kubernetes.io/projected/8f183a1c-3354-4637-891c-9bcbfb0daf67-istio-token\") pod \"8f183a1c-3354-4637-891c-9bcbfb0daf67\" (UID: \"8f183a1c-3354-4637-891c-9bcbfb0daf67\") " Nov 03 21:14:17.540966 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.540918 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f183a1c-3354-4637-891c-9bcbfb0daf67-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f183a1c-3354-4637-891c-9bcbfb0daf67-workload-certs\") pod \"8f183a1c-3354-4637-891c-9bcbfb0daf67\" (UID: \"8f183a1c-3354-4637-891c-9bcbfb0daf67\") " Nov 03 21:14:17.540966 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.540945 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f183a1c-3354-4637-891c-9bcbfb0daf67-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f183a1c-3354-4637-891c-9bcbfb0daf67-istio-data\") pod \"8f183a1c-3354-4637-891c-9bcbfb0daf67\" (UID: \"8f183a1c-3354-4637-891c-9bcbfb0daf67\") " Nov 03 21:14:17.541142 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.541064 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f183a1c-3354-4637-891c-9bcbfb0daf67-kube-api-access-sw8sr\" (UniqueName: \"kubernetes.io/projected/8f183a1c-3354-4637-891c-9bcbfb0daf67-kube-api-access-sw8sr\") pod \"8f183a1c-3354-4637-891c-9bcbfb0daf67\" (UID: \"8f183a1c-3354-4637-891c-9bcbfb0daf67\") " Nov 03 21:14:17.541142 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.541097 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8f183a1c-3354-4637-891c-9bcbfb0daf67-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f183a1c-3354-4637-891c-9bcbfb0daf67-istio-podinfo\") pod \"8f183a1c-3354-4637-891c-9bcbfb0daf67\" (UID: \"8f183a1c-3354-4637-891c-9bcbfb0daf67\") " Nov 03 21:14:17.541243 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.541157 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f183a1c-3354-4637-891c-9bcbfb0daf67-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f183a1c-3354-4637-891c-9bcbfb0daf67-workload-socket\") pod \"8f183a1c-3354-4637-891c-9bcbfb0daf67\" (UID: \"8f183a1c-3354-4637-891c-9bcbfb0daf67\") " Nov 03 21:14:17.541243 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.541206 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f183a1c-3354-4637-891c-9bcbfb0daf67-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f183a1c-3354-4637-891c-9bcbfb0daf67-credential-socket\") pod \"8f183a1c-3354-4637-891c-9bcbfb0daf67\" (UID: \"8f183a1c-3354-4637-891c-9bcbfb0daf67\") " Nov 03 21:14:17.541351 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.541253 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f183a1c-3354-4637-891c-9bcbfb0daf67-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f183a1c-3354-4637-891c-9bcbfb0daf67-istiod-ca-cert\") pod \"8f183a1c-3354-4637-891c-9bcbfb0daf67\" (UID: \"8f183a1c-3354-4637-891c-9bcbfb0daf67\") " Nov 03 21:14:17.541351 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.541326 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f183a1c-3354-4637-891c-9bcbfb0daf67-workload-certs" pod "8f183a1c-3354-4637-891c-9bcbfb0daf67" (UID: "8f183a1c-3354-4637-891c-9bcbfb0daf67"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:17.543136 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.542150 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f183a1c-3354-4637-891c-9bcbfb0daf67-workload-certs\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:17.543136 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.542482 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f183a1c-3354-4637-891c-9bcbfb0daf67-istio-data" pod "8f183a1c-3354-4637-891c-9bcbfb0daf67" (UID: "8f183a1c-3354-4637-891c-9bcbfb0daf67"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:17.543136 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.542829 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f183a1c-3354-4637-891c-9bcbfb0daf67-workload-socket" pod "8f183a1c-3354-4637-891c-9bcbfb0daf67" (UID: "8f183a1c-3354-4637-891c-9bcbfb0daf67"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:17.543518 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.543492 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f183a1c-3354-4637-891c-9bcbfb0daf67-credential-socket" pod "8f183a1c-3354-4637-891c-9bcbfb0daf67" (UID: "8f183a1c-3354-4637-891c-9bcbfb0daf67"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:17.543802 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.543757 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f183a1c-3354-4637-891c-9bcbfb0daf67-istiod-ca-cert" pod "8f183a1c-3354-4637-891c-9bcbfb0daf67" (UID: "8f183a1c-3354-4637-891c-9bcbfb0daf67"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:14:17.545332 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.544789 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f183a1c-3354-4637-891c-9bcbfb0daf67-istio-envoy" pod "8f183a1c-3354-4637-891c-9bcbfb0daf67" (UID: "8f183a1c-3354-4637-891c-9bcbfb0daf67"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:17.545332 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.544934 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f183a1c-3354-4637-891c-9bcbfb0daf67-istio-token" pod "8f183a1c-3354-4637-891c-9bcbfb0daf67" (UID: "8f183a1c-3354-4637-891c-9bcbfb0daf67"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:14:17.546546 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.546525 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f183a1c-3354-4637-891c-9bcbfb0daf67-kube-api-access-sw8sr" pod "8f183a1c-3354-4637-891c-9bcbfb0daf67" (UID: "8f183a1c-3354-4637-891c-9bcbfb0daf67"). InnerVolumeSpecName "kube-api-access-sw8sr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:14:17.547382 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.547357 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f183a1c-3354-4637-891c-9bcbfb0daf67-istio-podinfo" pod "8f183a1c-3354-4637-891c-9bcbfb0daf67" (UID: "8f183a1c-3354-4637-891c-9bcbfb0daf67"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:14:17.551382 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.551362 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd" Nov 03 21:14:17.559850 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.559824 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-x954c" Nov 03 21:14:17.563726 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.563702 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn" Nov 03 21:14:17.643314 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.643276 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2195a2fd-e42f-4e90-861d-a50b75f81973-kube-api-access-tftfp\" (UniqueName: \"kubernetes.io/projected/2195a2fd-e42f-4e90-861d-a50b75f81973-kube-api-access-tftfp\") pod \"2195a2fd-e42f-4e90-861d-a50b75f81973\" (UID: \"2195a2fd-e42f-4e90-861d-a50b75f81973\") " Nov 03 21:14:17.643554 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.643345 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2195a2fd-e42f-4e90-861d-a50b75f81973-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2195a2fd-e42f-4e90-861d-a50b75f81973-workload-socket\") pod \"2195a2fd-e42f-4e90-861d-a50b75f81973\" (UID: \"2195a2fd-e42f-4e90-861d-a50b75f81973\") " Nov 03 21:14:17.643554 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.643373 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2195a2fd-e42f-4e90-861d-a50b75f81973-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2195a2fd-e42f-4e90-861d-a50b75f81973-istio-envoy\") pod \"2195a2fd-e42f-4e90-861d-a50b75f81973\" (UID: \"2195a2fd-e42f-4e90-861d-a50b75f81973\") " Nov 03 21:14:17.643554 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.643407 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc-istio-podinfo\") pod \"fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc\" (UID: \"fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc\") " Nov 03 21:14:17.643554 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.643466 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c97844f3-937c-4873-865a-c408d83b0890-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c97844f3-937c-4873-865a-c408d83b0890-workload-socket\") pod \"c97844f3-937c-4873-865a-c408d83b0890\" (UID: \"c97844f3-937c-4873-865a-c408d83b0890\") " Nov 03 21:14:17.643554 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.643499 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c97844f3-937c-4873-865a-c408d83b0890-istio-token\" (UniqueName: \"kubernetes.io/projected/c97844f3-937c-4873-865a-c408d83b0890-istio-token\") pod \"c97844f3-937c-4873-865a-c408d83b0890\" (UID: \"c97844f3-937c-4873-865a-c408d83b0890\") " Nov 03 21:14:17.643554 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.643534 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2195a2fd-e42f-4e90-861d-a50b75f81973-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2195a2fd-e42f-4e90-861d-a50b75f81973-istio-podinfo\") pod \"2195a2fd-e42f-4e90-861d-a50b75f81973\" (UID: \"2195a2fd-e42f-4e90-861d-a50b75f81973\") " Nov 03 21:14:17.643899 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.643622 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc-istio-data\") pod \"fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc\" (UID: \"fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc\") " Nov 03 21:14:17.643899 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.643655 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c97844f3-937c-4873-865a-c408d83b0890-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c97844f3-937c-4873-865a-c408d83b0890-istio-podinfo\") pod \"c97844f3-937c-4873-865a-c408d83b0890\" (UID: \"c97844f3-937c-4873-865a-c408d83b0890\") " Nov 03 21:14:17.643899 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.643682 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c97844f3-937c-4873-865a-c408d83b0890-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c97844f3-937c-4873-865a-c408d83b0890-istiod-ca-cert\") pod \"c97844f3-937c-4873-865a-c408d83b0890\" (UID: \"c97844f3-937c-4873-865a-c408d83b0890\") " Nov 03 21:14:17.643899 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.643725 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2195a2fd-e42f-4e90-861d-a50b75f81973-istio-token\" (UniqueName: \"kubernetes.io/projected/2195a2fd-e42f-4e90-861d-a50b75f81973-istio-token\") pod \"2195a2fd-e42f-4e90-861d-a50b75f81973\" (UID: \"2195a2fd-e42f-4e90-861d-a50b75f81973\") " Nov 03 21:14:17.643899 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.643726 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2195a2fd-e42f-4e90-861d-a50b75f81973-workload-socket" pod "2195a2fd-e42f-4e90-861d-a50b75f81973" (UID: "2195a2fd-e42f-4e90-861d-a50b75f81973"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:17.643899 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.643750 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc-credential-socket\") pod \"fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc\" (UID: \"fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc\") " Nov 03 21:14:17.643899 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.643803 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc-workload-socket\") pod \"fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc\" (UID: \"fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc\") " Nov 03 21:14:17.643899 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.643847 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc-workload-certs\") pod \"fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc\" (UID: \"fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc\") " Nov 03 21:14:17.643899 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.643882 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc-istio-token\" (UniqueName: \"kubernetes.io/projected/fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc-istio-token\") pod \"fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc\" (UID: \"fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc\") " Nov 03 21:14:17.644407 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.643980 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c97844f3-937c-4873-865a-c408d83b0890-kube-api-access-sdrmz\" (UniqueName: \"kubernetes.io/projected/c97844f3-937c-4873-865a-c408d83b0890-kube-api-access-sdrmz\") pod \"c97844f3-937c-4873-865a-c408d83b0890\" (UID: \"c97844f3-937c-4873-865a-c408d83b0890\") " Nov 03 21:14:17.644407 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.644003 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc-credential-socket" pod "fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc" (UID: "fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:17.644407 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.644037 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc-istio-envoy\") pod \"fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc\" (UID: \"fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc\") " Nov 03 21:14:17.644407 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.644066 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2195a2fd-e42f-4e90-861d-a50b75f81973-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2195a2fd-e42f-4e90-861d-a50b75f81973-workload-certs\") pod \"2195a2fd-e42f-4e90-861d-a50b75f81973\" (UID: \"2195a2fd-e42f-4e90-861d-a50b75f81973\") " Nov 03 21:14:17.644407 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.644113 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2195a2fd-e42f-4e90-861d-a50b75f81973-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2195a2fd-e42f-4e90-861d-a50b75f81973-istiod-ca-cert\") pod \"2195a2fd-e42f-4e90-861d-a50b75f81973\" (UID: \"2195a2fd-e42f-4e90-861d-a50b75f81973\") " Nov 03 21:14:17.644407 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.644103 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c97844f3-937c-4873-865a-c408d83b0890-workload-socket" pod "c97844f3-937c-4873-865a-c408d83b0890" (UID: "c97844f3-937c-4873-865a-c408d83b0890"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:17.644407 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.644123 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc-workload-socket" pod "fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc" (UID: "fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:17.644407 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.644192 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc-istio-data" pod "fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc" (UID: "fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:17.644407 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.644224 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c97844f3-937c-4873-865a-c408d83b0890-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c97844f3-937c-4873-865a-c408d83b0890-credential-socket\") pod \"c97844f3-937c-4873-865a-c408d83b0890\" (UID: \"c97844f3-937c-4873-865a-c408d83b0890\") " Nov 03 21:14:17.644407 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.644259 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc-kube-api-access-474mp\" (UniqueName: \"kubernetes.io/projected/fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc-kube-api-access-474mp\") pod \"fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc\" (UID: \"fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc\") " Nov 03 21:14:17.644407 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.644290 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c97844f3-937c-4873-865a-c408d83b0890-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c97844f3-937c-4873-865a-c408d83b0890-workload-certs\") pod \"c97844f3-937c-4873-865a-c408d83b0890\" (UID: \"c97844f3-937c-4873-865a-c408d83b0890\") " Nov 03 21:14:17.644407 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.644322 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c97844f3-937c-4873-865a-c408d83b0890-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c97844f3-937c-4873-865a-c408d83b0890-istio-envoy\") pod \"c97844f3-937c-4873-865a-c408d83b0890\" (UID: \"c97844f3-937c-4873-865a-c408d83b0890\") " Nov 03 21:14:17.644407 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.644349 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc-istiod-ca-cert\") pod \"fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc\" (UID: \"fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc\") " Nov 03 21:14:17.644407 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.644384 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2195a2fd-e42f-4e90-861d-a50b75f81973-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2195a2fd-e42f-4e90-861d-a50b75f81973-credential-socket\") pod \"2195a2fd-e42f-4e90-861d-a50b75f81973\" (UID: \"2195a2fd-e42f-4e90-861d-a50b75f81973\") " Nov 03 21:14:17.645483 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.644416 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2195a2fd-e42f-4e90-861d-a50b75f81973-istiod-ca-cert" pod "2195a2fd-e42f-4e90-861d-a50b75f81973" (UID: "2195a2fd-e42f-4e90-861d-a50b75f81973"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:14:17.645483 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.644456 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2195a2fd-e42f-4e90-861d-a50b75f81973-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2195a2fd-e42f-4e90-861d-a50b75f81973-istio-data\") pod \"2195a2fd-e42f-4e90-861d-a50b75f81973\" (UID: \"2195a2fd-e42f-4e90-861d-a50b75f81973\") " Nov 03 21:14:17.645483 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.644484 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c97844f3-937c-4873-865a-c408d83b0890-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c97844f3-937c-4873-865a-c408d83b0890-istio-data\") pod \"c97844f3-937c-4873-865a-c408d83b0890\" (UID: \"c97844f3-937c-4873-865a-c408d83b0890\") " Nov 03 21:14:17.645483 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.645278 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc-workload-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:17.645483 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.645301 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f183a1c-3354-4637-891c-9bcbfb0daf67-istio-envoy\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:17.645483 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.645317 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f183a1c-3354-4637-891c-9bcbfb0daf67-istio-token\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:17.645483 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.645331 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f183a1c-3354-4637-891c-9bcbfb0daf67-istio-data\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:17.645483 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.645347 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2195a2fd-e42f-4e90-861d-a50b75f81973-istiod-ca-cert\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:17.645483 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.645362 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sw8sr\" (UniqueName: \"kubernetes.io/projected/8f183a1c-3354-4637-891c-9bcbfb0daf67-kube-api-access-sw8sr\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:17.645483 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.645379 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f183a1c-3354-4637-891c-9bcbfb0daf67-istio-podinfo\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:17.645483 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.645393 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2195a2fd-e42f-4e90-861d-a50b75f81973-workload-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:17.645483 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.645409 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f183a1c-3354-4637-891c-9bcbfb0daf67-workload-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:17.645483 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.645424 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c97844f3-937c-4873-865a-c408d83b0890-workload-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:17.645483 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.645458 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f183a1c-3354-4637-891c-9bcbfb0daf67-credential-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:17.645483 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.645475 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f183a1c-3354-4637-891c-9bcbfb0daf67-istiod-ca-cert\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:17.645483 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.645489 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc-istio-data\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:17.646468 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.645504 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc-credential-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:17.646468 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.645746 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c97844f3-937c-4873-865a-c408d83b0890-istio-data" pod "c97844f3-937c-4873-865a-c408d83b0890" (UID: "c97844f3-937c-4873-865a-c408d83b0890"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:17.646468 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.646150 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc-workload-certs" pod "fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc" (UID: "fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:17.646651 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.646483 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2195a2fd-e42f-4e90-861d-a50b75f81973-workload-certs" pod "2195a2fd-e42f-4e90-861d-a50b75f81973" (UID: "2195a2fd-e42f-4e90-861d-a50b75f81973"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:17.647171 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.647125 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc-istiod-ca-cert" pod "fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc" (UID: "fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:14:17.647171 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.647131 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc-istio-podinfo" pod "fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc" (UID: "fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:14:17.647171 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.646926 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c97844f3-937c-4873-865a-c408d83b0890-credential-socket" pod "c97844f3-937c-4873-865a-c408d83b0890" (UID: "c97844f3-937c-4873-865a-c408d83b0890"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:17.647560 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.647293 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2195a2fd-e42f-4e90-861d-a50b75f81973-istio-data" pod "2195a2fd-e42f-4e90-861d-a50b75f81973" (UID: "2195a2fd-e42f-4e90-861d-a50b75f81973"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:17.647560 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.647197 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2195a2fd-e42f-4e90-861d-a50b75f81973-kube-api-access-tftfp" pod "2195a2fd-e42f-4e90-861d-a50b75f81973" (UID: "2195a2fd-e42f-4e90-861d-a50b75f81973"). InnerVolumeSpecName "kube-api-access-tftfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:14:17.647560 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.647356 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2195a2fd-e42f-4e90-861d-a50b75f81973-credential-socket" pod "2195a2fd-e42f-4e90-861d-a50b75f81973" (UID: "2195a2fd-e42f-4e90-861d-a50b75f81973"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:17.647560 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.647527 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c97844f3-937c-4873-865a-c408d83b0890-kube-api-access-sdrmz" pod "c97844f3-937c-4873-865a-c408d83b0890" (UID: "c97844f3-937c-4873-865a-c408d83b0890"). InnerVolumeSpecName "kube-api-access-sdrmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:14:17.647883 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.647863 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc-istio-envoy" pod "fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc" (UID: "fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:17.648511 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.648475 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c97844f3-937c-4873-865a-c408d83b0890-workload-certs" pod "c97844f3-937c-4873-865a-c408d83b0890" (UID: "c97844f3-937c-4873-865a-c408d83b0890"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:17.648615 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.648523 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c97844f3-937c-4873-865a-c408d83b0890-istiod-ca-cert" pod "c97844f3-937c-4873-865a-c408d83b0890" (UID: "c97844f3-937c-4873-865a-c408d83b0890"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:14:17.649154 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.649128 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc-istio-token" pod "fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc" (UID: "fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:14:17.649255 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.649186 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2195a2fd-e42f-4e90-861d-a50b75f81973-istio-podinfo" pod "2195a2fd-e42f-4e90-861d-a50b75f81973" (UID: "2195a2fd-e42f-4e90-861d-a50b75f81973"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:14:17.649342 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.649313 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2195a2fd-e42f-4e90-861d-a50b75f81973-istio-envoy" pod "2195a2fd-e42f-4e90-861d-a50b75f81973" (UID: "2195a2fd-e42f-4e90-861d-a50b75f81973"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:17.649495 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.649473 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c97844f3-937c-4873-865a-c408d83b0890-istio-podinfo" pod "c97844f3-937c-4873-865a-c408d83b0890" (UID: "c97844f3-937c-4873-865a-c408d83b0890"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:14:17.649607 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.649589 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc-kube-api-access-474mp" pod "fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc" (UID: "fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc"). InnerVolumeSpecName "kube-api-access-474mp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:14:17.649711 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.649692 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2195a2fd-e42f-4e90-861d-a50b75f81973-istio-token" pod "2195a2fd-e42f-4e90-861d-a50b75f81973" (UID: "2195a2fd-e42f-4e90-861d-a50b75f81973"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:14:17.650426 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.650410 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c97844f3-937c-4873-865a-c408d83b0890-istio-envoy" pod "c97844f3-937c-4873-865a-c408d83b0890" (UID: "c97844f3-937c-4873-865a-c408d83b0890"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:17.650589 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.650572 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c97844f3-937c-4873-865a-c408d83b0890-istio-token" pod "c97844f3-937c-4873-865a-c408d83b0890" (UID: "c97844f3-937c-4873-865a-c408d83b0890"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:14:17.714692 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.714656 4305 generic.go:356] "Generic (PLEG): container finished" podID="c97844f3-937c-4873-865a-c408d83b0890" containerID="3c92e1c0e4ef243987942b115cada19a70ea9c7b130d49dc0a0f47344c6987bd" exitCode=0 Nov 03 21:14:17.714902 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.714781 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd" Nov 03 21:14:17.714902 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.714789 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd" event={"ID":"c97844f3-937c-4873-865a-c408d83b0890","Type":"ContainerDied","Data":"3c92e1c0e4ef243987942b115cada19a70ea9c7b130d49dc0a0f47344c6987bd"} Nov 03 21:14:17.714902 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.714882 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd" event={"ID":"c97844f3-937c-4873-865a-c408d83b0890","Type":"ContainerDied","Data":"f6f792c21e6a72d1a7fc89f90ea85956f50744101de86df2e1bcc1f4c8b28eeb"} Nov 03 21:14:17.715072 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.714913 4305 scope.go:122] "RemoveContainer" containerID="3c92e1c0e4ef243987942b115cada19a70ea9c7b130d49dc0a0f47344c6987bd" Nov 03 21:14:17.716538 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.716514 4305 generic.go:356] "Generic (PLEG): container finished" podID="8f183a1c-3354-4637-891c-9bcbfb0daf67" containerID="134d2d0e29ddca31f2a1783a064d951bb0fb94713b9c2e4ed036e16bda28bdb2" exitCode=0 Nov 03 21:14:17.716691 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.716589 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-27ssq" event={"ID":"8f183a1c-3354-4637-891c-9bcbfb0daf67","Type":"ContainerDied","Data":"134d2d0e29ddca31f2a1783a064d951bb0fb94713b9c2e4ed036e16bda28bdb2"} Nov 03 21:14:17.716691 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.716632 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-27ssq" event={"ID":"8f183a1c-3354-4637-891c-9bcbfb0daf67","Type":"ContainerDied","Data":"9802010118d597136349f8ddc1a8f838963ef52d80dcadef5cc2dbbe402b5a29"} Nov 03 21:14:17.716824 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.716768 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-27ssq" Nov 03 21:14:17.719827 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.719761 4305 generic.go:356] "Generic (PLEG): container finished" podID="2195a2fd-e42f-4e90-861d-a50b75f81973" containerID="38753558348ff64f5e879a4041a51532b05251bd1d16f6c0200cca6e992cd8ef" exitCode=0 Nov 03 21:14:17.719955 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.719847 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-x954c" event={"ID":"2195a2fd-e42f-4e90-861d-a50b75f81973","Type":"ContainerDied","Data":"38753558348ff64f5e879a4041a51532b05251bd1d16f6c0200cca6e992cd8ef"} Nov 03 21:14:17.719955 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.719891 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-x954c" event={"ID":"2195a2fd-e42f-4e90-861d-a50b75f81973","Type":"ContainerDied","Data":"fbb1d10d456b297a6ca33e467e8aad517bed14ef028a8132bbbe3caec5b6a31c"} Nov 03 21:14:17.720078 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.719990 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-x954c" Nov 03 21:14:17.727609 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.727051 4305 generic.go:356] "Generic (PLEG): container finished" podID="fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc" containerID="32dca95a4802050f05da86005a6ea74031dc7b0017bbe0276b03e076fefd6383" exitCode=0 Nov 03 21:14:17.727609 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.727243 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn" event={"ID":"fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc","Type":"ContainerDied","Data":"32dca95a4802050f05da86005a6ea74031dc7b0017bbe0276b03e076fefd6383"} Nov 03 21:14:17.727609 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.727288 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn" event={"ID":"fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc","Type":"ContainerDied","Data":"6bd5edf0e0cf9bf7ecd638e4ec2a1ffc6ed5f24f58e6bcfe1e2df584b8f028c3"} Nov 03 21:14:17.727609 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.727368 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn" Nov 03 21:14:17.733671 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.733643 4305 scope.go:122] "RemoveContainer" containerID="71272b9187b8c275851e3b0e36a1ba0cbbf1f181207413e4bf02904c4edd0d07" Nov 03 21:14:17.744233 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.744208 4305 scope.go:122] "RemoveContainer" containerID="3c92e1c0e4ef243987942b115cada19a70ea9c7b130d49dc0a0f47344c6987bd" Nov 03 21:14:17.744605 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:14:17.744582 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c92e1c0e4ef243987942b115cada19a70ea9c7b130d49dc0a0f47344c6987bd\": container with ID starting with 3c92e1c0e4ef243987942b115cada19a70ea9c7b130d49dc0a0f47344c6987bd not found: ID does not exist" containerID="3c92e1c0e4ef243987942b115cada19a70ea9c7b130d49dc0a0f47344c6987bd" Nov 03 21:14:17.744716 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.744619 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c92e1c0e4ef243987942b115cada19a70ea9c7b130d49dc0a0f47344c6987bd"} err="failed to get container status \"3c92e1c0e4ef243987942b115cada19a70ea9c7b130d49dc0a0f47344c6987bd\": rpc error: code = NotFound desc = could not find container \"3c92e1c0e4ef243987942b115cada19a70ea9c7b130d49dc0a0f47344c6987bd\": container with ID starting with 3c92e1c0e4ef243987942b115cada19a70ea9c7b130d49dc0a0f47344c6987bd not found: ID does not exist" Nov 03 21:14:17.744716 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.744645 4305 scope.go:122] "RemoveContainer" containerID="71272b9187b8c275851e3b0e36a1ba0cbbf1f181207413e4bf02904c4edd0d07" Nov 03 21:14:17.744934 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:14:17.744912 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71272b9187b8c275851e3b0e36a1ba0cbbf1f181207413e4bf02904c4edd0d07\": container with ID starting with 71272b9187b8c275851e3b0e36a1ba0cbbf1f181207413e4bf02904c4edd0d07 not found: ID does not exist" containerID="71272b9187b8c275851e3b0e36a1ba0cbbf1f181207413e4bf02904c4edd0d07" Nov 03 21:14:17.744991 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.744942 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71272b9187b8c275851e3b0e36a1ba0cbbf1f181207413e4bf02904c4edd0d07"} err="failed to get container status \"71272b9187b8c275851e3b0e36a1ba0cbbf1f181207413e4bf02904c4edd0d07\": rpc error: code = NotFound desc = could not find container \"71272b9187b8c275851e3b0e36a1ba0cbbf1f181207413e4bf02904c4edd0d07\": container with ID starting with 71272b9187b8c275851e3b0e36a1ba0cbbf1f181207413e4bf02904c4edd0d07 not found: ID does not exist" Nov 03 21:14:17.744991 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.744960 4305 scope.go:122] "RemoveContainer" containerID="134d2d0e29ddca31f2a1783a064d951bb0fb94713b9c2e4ed036e16bda28bdb2" Nov 03 21:14:17.746102 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.746082 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c97844f3-937c-4873-865a-c408d83b0890-credential-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:17.746171 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.746104 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-474mp\" (UniqueName: \"kubernetes.io/projected/fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc-kube-api-access-474mp\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:17.746171 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.746114 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c97844f3-937c-4873-865a-c408d83b0890-workload-certs\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:17.746171 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.746124 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c97844f3-937c-4873-865a-c408d83b0890-istio-envoy\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:17.746171 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.746132 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc-istiod-ca-cert\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:17.746171 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.746140 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2195a2fd-e42f-4e90-861d-a50b75f81973-credential-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:17.746171 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.746148 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2195a2fd-e42f-4e90-861d-a50b75f81973-istio-data\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:17.746171 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.746156 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c97844f3-937c-4873-865a-c408d83b0890-istio-data\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:17.746171 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.746164 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tftfp\" (UniqueName: \"kubernetes.io/projected/2195a2fd-e42f-4e90-861d-a50b75f81973-kube-api-access-tftfp\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:17.746495 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.746174 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2195a2fd-e42f-4e90-861d-a50b75f81973-istio-envoy\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:17.746495 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.746182 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc-istio-podinfo\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:17.746495 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.746191 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c97844f3-937c-4873-865a-c408d83b0890-istio-token\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:17.746495 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.746199 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2195a2fd-e42f-4e90-861d-a50b75f81973-istio-podinfo\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:17.746495 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.746207 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c97844f3-937c-4873-865a-c408d83b0890-istio-podinfo\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:17.746495 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.746215 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c97844f3-937c-4873-865a-c408d83b0890-istiod-ca-cert\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:17.746495 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.746223 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2195a2fd-e42f-4e90-861d-a50b75f81973-istio-token\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:17.746495 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.746230 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc-workload-certs\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:17.746495 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.746238 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc-istio-token\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:17.746495 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.746247 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdrmz\" (UniqueName: \"kubernetes.io/projected/c97844f3-937c-4873-865a-c408d83b0890-kube-api-access-sdrmz\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:17.746495 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.746255 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc-istio-envoy\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:17.746495 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.746263 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2195a2fd-e42f-4e90-861d-a50b75f81973-workload-certs\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:17.757093 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.757071 4305 scope.go:122] "RemoveContainer" containerID="134d2d0e29ddca31f2a1783a064d951bb0fb94713b9c2e4ed036e16bda28bdb2" Nov 03 21:14:17.757444 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:14:17.757412 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"134d2d0e29ddca31f2a1783a064d951bb0fb94713b9c2e4ed036e16bda28bdb2\": container with ID starting with 134d2d0e29ddca31f2a1783a064d951bb0fb94713b9c2e4ed036e16bda28bdb2 not found: ID does not exist" containerID="134d2d0e29ddca31f2a1783a064d951bb0fb94713b9c2e4ed036e16bda28bdb2" Nov 03 21:14:17.757514 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.757456 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"134d2d0e29ddca31f2a1783a064d951bb0fb94713b9c2e4ed036e16bda28bdb2"} err="failed to get container status \"134d2d0e29ddca31f2a1783a064d951bb0fb94713b9c2e4ed036e16bda28bdb2\": rpc error: code = NotFound desc = could not find container \"134d2d0e29ddca31f2a1783a064d951bb0fb94713b9c2e4ed036e16bda28bdb2\": container with ID starting with 134d2d0e29ddca31f2a1783a064d951bb0fb94713b9c2e4ed036e16bda28bdb2 not found: ID does not exist" Nov 03 21:14:17.757514 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.757475 4305 scope.go:122] "RemoveContainer" containerID="38753558348ff64f5e879a4041a51532b05251bd1d16f6c0200cca6e992cd8ef" Nov 03 21:14:17.768716 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.768694 4305 scope.go:122] "RemoveContainer" containerID="38753558348ff64f5e879a4041a51532b05251bd1d16f6c0200cca6e992cd8ef" Nov 03 21:14:17.769039 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:14:17.769016 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38753558348ff64f5e879a4041a51532b05251bd1d16f6c0200cca6e992cd8ef\": container with ID starting with 38753558348ff64f5e879a4041a51532b05251bd1d16f6c0200cca6e992cd8ef not found: ID does not exist" containerID="38753558348ff64f5e879a4041a51532b05251bd1d16f6c0200cca6e992cd8ef" Nov 03 21:14:17.769130 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.769052 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38753558348ff64f5e879a4041a51532b05251bd1d16f6c0200cca6e992cd8ef"} err="failed to get container status \"38753558348ff64f5e879a4041a51532b05251bd1d16f6c0200cca6e992cd8ef\": rpc error: code = NotFound desc = could not find container \"38753558348ff64f5e879a4041a51532b05251bd1d16f6c0200cca6e992cd8ef\": container with ID starting with 38753558348ff64f5e879a4041a51532b05251bd1d16f6c0200cca6e992cd8ef not found: ID does not exist" Nov 03 21:14:17.769130 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.769076 4305 scope.go:122] "RemoveContainer" containerID="32dca95a4802050f05da86005a6ea74031dc7b0017bbe0276b03e076fefd6383" Nov 03 21:14:17.781278 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.781254 4305 scope.go:122] "RemoveContainer" containerID="32dca95a4802050f05da86005a6ea74031dc7b0017bbe0276b03e076fefd6383" Nov 03 21:14:17.781672 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:14:17.781647 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32dca95a4802050f05da86005a6ea74031dc7b0017bbe0276b03e076fefd6383\": container with ID starting with 32dca95a4802050f05da86005a6ea74031dc7b0017bbe0276b03e076fefd6383 not found: ID does not exist" containerID="32dca95a4802050f05da86005a6ea74031dc7b0017bbe0276b03e076fefd6383" Nov 03 21:14:17.781792 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:17.781680 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32dca95a4802050f05da86005a6ea74031dc7b0017bbe0276b03e076fefd6383"} err="failed to get container status \"32dca95a4802050f05da86005a6ea74031dc7b0017bbe0276b03e076fefd6383\": rpc error: code = NotFound desc = could not find container \"32dca95a4802050f05da86005a6ea74031dc7b0017bbe0276b03e076fefd6383\": container with ID starting with 32dca95a4802050f05da86005a6ea74031dc7b0017bbe0276b03e076fefd6383 not found: ID does not exist" Nov 03 21:14:18.141751 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.141662 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd"] Nov 03 21:14:18.147336 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.147302 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-jkfsd"] Nov 03 21:14:18.173773 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.173730 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-12a423-559b96f96d-nmf2b" podUID="3e6b8d49-0db7-4bbc-9b3e-1105d2bdbc78" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:14:18.173773 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.173760 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-12a423-846d4b6867-grwcg" podUID="7b0135ee-b547-46e8-9953-6a46cb61f2c4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:14:18.255926 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.255877 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-862138-65fb688c66-fx4kk"] Nov 03 21:14:18.259740 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.259712 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-27ssq"] Nov 03 21:14:18.262543 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.262517 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-27ssq"] Nov 03 21:14:18.374486 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.374414 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-383e06ff-tzdkz"] Nov 03 21:14:18.374981 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.374965 4305 cpu_manager.go:414] "containerMap: removing container" podUID="2195a2fd-e42f-4e90-861d-a50b75f81973" containerName="istio-proxy" Nov 03 21:14:18.375027 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.374985 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2195a2fd-e42f-4e90-861d-a50b75f81973" containerName="istio-proxy" Nov 03 21:14:18.375027 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.375003 4305 cpu_manager.go:414] "containerMap: removing container" podUID="fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc" containerName="istio-proxy" Nov 03 21:14:18.375027 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.375008 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc" containerName="istio-proxy" Nov 03 21:14:18.375027 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.375023 4305 cpu_manager.go:414] "containerMap: removing container" podUID="c97844f3-937c-4873-865a-c408d83b0890" containerName="istio-proxy" Nov 03 21:14:18.375158 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.375029 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c97844f3-937c-4873-865a-c408d83b0890" containerName="istio-proxy" Nov 03 21:14:18.375158 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.375034 4305 cpu_manager.go:414] "containerMap: removing container" podUID="da5ca27e-9637-4eea-b462-06f298f367de" containerName="revoke-keys" Nov 03 21:14:18.375158 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.375040 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da5ca27e-9637-4eea-b462-06f298f367de" containerName="revoke-keys" Nov 03 21:14:18.375158 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.375048 4305 cpu_manager.go:414] "containerMap: removing container" podUID="7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3" containerName="main" Nov 03 21:14:18.375158 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.375053 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3" containerName="main" Nov 03 21:14:18.375158 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.375060 4305 cpu_manager.go:414] "containerMap: removing container" podUID="239c6c9d-3ac5-49b5-9093-e686f002ce2a" containerName="istio-proxy" Nov 03 21:14:18.375158 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.375065 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="239c6c9d-3ac5-49b5-9093-e686f002ce2a" containerName="istio-proxy" Nov 03 21:14:18.375158 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.375075 4305 cpu_manager.go:414] "containerMap: removing container" podUID="239c6c9d-3ac5-49b5-9093-e686f002ce2a" containerName="istio-proxy" Nov 03 21:14:18.375158 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.375079 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="239c6c9d-3ac5-49b5-9093-e686f002ce2a" containerName="istio-proxy" Nov 03 21:14:18.375158 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.375085 4305 cpu_manager.go:414] "containerMap: removing container" podUID="63a704b4-51ce-46c3-8007-34b308aff332" containerName="main" Nov 03 21:14:18.375158 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.375089 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63a704b4-51ce-46c3-8007-34b308aff332" containerName="main" Nov 03 21:14:18.375158 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.375096 4305 cpu_manager.go:414] "containerMap: removing container" podUID="a568044d-1a39-4916-86ea-fadfbcb6ddcd" containerName="revoke-keys" Nov 03 21:14:18.375158 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.375100 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a568044d-1a39-4916-86ea-fadfbcb6ddcd" containerName="revoke-keys" Nov 03 21:14:18.375158 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.375111 4305 cpu_manager.go:414] "containerMap: removing container" podUID="8f183a1c-3354-4637-891c-9bcbfb0daf67" containerName="istio-proxy" Nov 03 21:14:18.375158 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.375115 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f183a1c-3354-4637-891c-9bcbfb0daf67" containerName="istio-proxy" Nov 03 21:14:18.375620 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.375176 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc" containerName="istio-proxy" Nov 03 21:14:18.375620 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.375186 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="c97844f3-937c-4873-865a-c408d83b0890" containerName="istio-proxy" Nov 03 21:14:18.375620 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.375192 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="da5ca27e-9637-4eea-b462-06f298f367de" containerName="revoke-keys" Nov 03 21:14:18.375620 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.375198 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c05fc7b-83f3-43d0-830b-f4fc2e9fadb3" containerName="main" Nov 03 21:14:18.375620 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.375205 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="239c6c9d-3ac5-49b5-9093-e686f002ce2a" containerName="istio-proxy" Nov 03 21:14:18.375620 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.375211 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="239c6c9d-3ac5-49b5-9093-e686f002ce2a" containerName="istio-proxy" Nov 03 21:14:18.375620 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.375217 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="63a704b4-51ce-46c3-8007-34b308aff332" containerName="main" Nov 03 21:14:18.375620 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.375222 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="a568044d-1a39-4916-86ea-fadfbcb6ddcd" containerName="revoke-keys" Nov 03 21:14:18.375620 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.375228 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f183a1c-3354-4637-891c-9bcbfb0daf67" containerName="istio-proxy" Nov 03 21:14:18.375620 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.375235 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="2195a2fd-e42f-4e90-861d-a50b75f81973" containerName="istio-proxy" Nov 03 21:14:18.379309 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.379280 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-383e06ff-tzdkz" Nov 03 21:14:18.380838 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.380813 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-x954c"] Nov 03 21:14:18.385732 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.385696 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-x954c"] Nov 03 21:14:18.388559 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.388536 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-862138-6c8d9c5567-7nvj6"] Nov 03 21:14:18.429271 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.429231 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adfa0d96-rqh4f"] Nov 03 21:14:18.429767 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.429752 4305 cpu_manager.go:414] "containerMap: removing container" podUID="c97844f3-937c-4873-865a-c408d83b0890" containerName="istio-proxy" Nov 03 21:14:18.429825 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.429769 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c97844f3-937c-4873-865a-c408d83b0890" containerName="istio-proxy" Nov 03 21:14:18.429893 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.429855 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="c97844f3-937c-4873-865a-c408d83b0890" containerName="istio-proxy" Nov 03 21:14:18.432091 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.432064 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77vtd\"" Nov 03 21:14:18.433290 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.433263 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adfa0d96-rqh4f" Nov 03 21:14:18.434534 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.434511 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:14:18.435336 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.435318 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:14:18.435489 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.435349 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:14:18.435489 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.435420 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-383e06ff-tzdkz"] Nov 03 21:14:18.453888 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.453858 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a20ce38b-39a2-4e14-92f9-ad37dd2ad124-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-383e06ff-tzdkz\" (UID: \"a20ce38b-39a2-4e14-92f9-ad37dd2ad124\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-383e06ff-tzdkz" Nov 03 21:14:18.454067 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.454044 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a20ce38b-39a2-4e14-92f9-ad37dd2ad124-maas-api-xks-ca\") pod \"maas-api-revoke-sub-383e06ff-tzdkz\" (UID: \"a20ce38b-39a2-4e14-92f9-ad37dd2ad124\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-383e06ff-tzdkz" Nov 03 21:14:18.454124 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.454096 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvl9f\" (UniqueName: \"kubernetes.io/projected/a20ce38b-39a2-4e14-92f9-ad37dd2ad124-kube-api-access-vvl9f\") pod \"maas-api-revoke-sub-383e06ff-tzdkz\" (UID: \"a20ce38b-39a2-4e14-92f9-ad37dd2ad124\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-383e06ff-tzdkz" Nov 03 21:14:18.495443 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.495397 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adfa0d96-rqh4f"] Nov 03 21:14:18.499912 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.499761 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn"] Nov 03 21:14:18.506271 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.506237 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jmpn"] Nov 03 21:14:18.555512 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.555476 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2e9589d-058b-47fb-a28d-9b92a855c9ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-adfa0d96-rqh4f\" (UID: \"a2e9589d-058b-47fb-a28d-9b92a855c9ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adfa0d96-rqh4f" Nov 03 21:14:18.555680 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.555525 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsdfm\" (UniqueName: \"kubernetes.io/projected/a2e9589d-058b-47fb-a28d-9b92a855c9ca-kube-api-access-rsdfm\") pod \"maas-api-revoke-sub-adfa0d96-rqh4f\" (UID: \"a2e9589d-058b-47fb-a28d-9b92a855c9ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adfa0d96-rqh4f" Nov 03 21:14:18.555680 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.555598 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a20ce38b-39a2-4e14-92f9-ad37dd2ad124-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-383e06ff-tzdkz\" (UID: \"a20ce38b-39a2-4e14-92f9-ad37dd2ad124\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-383e06ff-tzdkz" Nov 03 21:14:18.555680 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.555617 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2e9589d-058b-47fb-a28d-9b92a855c9ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-adfa0d96-rqh4f\" (UID: \"a2e9589d-058b-47fb-a28d-9b92a855c9ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adfa0d96-rqh4f" Nov 03 21:14:18.555822 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.555691 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a20ce38b-39a2-4e14-92f9-ad37dd2ad124-maas-api-xks-ca\") pod \"maas-api-revoke-sub-383e06ff-tzdkz\" (UID: \"a20ce38b-39a2-4e14-92f9-ad37dd2ad124\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-383e06ff-tzdkz" Nov 03 21:14:18.555822 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.555720 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvl9f\" (UniqueName: \"kubernetes.io/projected/a20ce38b-39a2-4e14-92f9-ad37dd2ad124-kube-api-access-vvl9f\") pod \"maas-api-revoke-sub-383e06ff-tzdkz\" (UID: \"a20ce38b-39a2-4e14-92f9-ad37dd2ad124\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-383e06ff-tzdkz" Nov 03 21:14:18.556350 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.556329 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a20ce38b-39a2-4e14-92f9-ad37dd2ad124-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-383e06ff-tzdkz\" (UID: \"a20ce38b-39a2-4e14-92f9-ad37dd2ad124\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-383e06ff-tzdkz" Nov 03 21:14:18.558291 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.558269 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a20ce38b-39a2-4e14-92f9-ad37dd2ad124-maas-api-xks-ca\") pod \"maas-api-revoke-sub-383e06ff-tzdkz\" (UID: \"a20ce38b-39a2-4e14-92f9-ad37dd2ad124\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-383e06ff-tzdkz" Nov 03 21:14:18.616764 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.616733 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvl9f\" (UniqueName: \"kubernetes.io/projected/a20ce38b-39a2-4e14-92f9-ad37dd2ad124-kube-api-access-vvl9f\") pod \"maas-api-revoke-sub-383e06ff-tzdkz\" (UID: \"a20ce38b-39a2-4e14-92f9-ad37dd2ad124\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-383e06ff-tzdkz" Nov 03 21:14:18.656358 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.656321 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2e9589d-058b-47fb-a28d-9b92a855c9ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-adfa0d96-rqh4f\" (UID: \"a2e9589d-058b-47fb-a28d-9b92a855c9ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adfa0d96-rqh4f" Nov 03 21:14:18.656358 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.656361 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsdfm\" (UniqueName: \"kubernetes.io/projected/a2e9589d-058b-47fb-a28d-9b92a855c9ca-kube-api-access-rsdfm\") pod \"maas-api-revoke-sub-adfa0d96-rqh4f\" (UID: \"a2e9589d-058b-47fb-a28d-9b92a855c9ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adfa0d96-rqh4f" Nov 03 21:14:18.656667 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.656406 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2e9589d-058b-47fb-a28d-9b92a855c9ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-adfa0d96-rqh4f\" (UID: \"a2e9589d-058b-47fb-a28d-9b92a855c9ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adfa0d96-rqh4f" Nov 03 21:14:18.657057 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.657029 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2e9589d-058b-47fb-a28d-9b92a855c9ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-adfa0d96-rqh4f\" (UID: \"a2e9589d-058b-47fb-a28d-9b92a855c9ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adfa0d96-rqh4f" Nov 03 21:14:18.659372 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.659344 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2e9589d-058b-47fb-a28d-9b92a855c9ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-adfa0d96-rqh4f\" (UID: \"a2e9589d-058b-47fb-a28d-9b92a855c9ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adfa0d96-rqh4f" Nov 03 21:14:18.696925 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.696831 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-383e06ff-tzdkz" Nov 03 21:14:18.717592 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.717556 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsdfm\" (UniqueName: \"kubernetes.io/projected/a2e9589d-058b-47fb-a28d-9b92a855c9ca-kube-api-access-rsdfm\") pod \"maas-api-revoke-sub-adfa0d96-rqh4f\" (UID: \"a2e9589d-058b-47fb-a28d-9b92a855c9ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adfa0d96-rqh4f" Nov 03 21:14:18.757183 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.757139 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adfa0d96-rqh4f" Nov 03 21:14:18.896001 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.895965 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2195a2fd-e42f-4e90-861d-a50b75f81973" path="/var/lib/kubelet/pods/2195a2fd-e42f-4e90-861d-a50b75f81973/volumes" Nov 03 21:14:18.896473 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.896456 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f183a1c-3354-4637-891c-9bcbfb0daf67" path="/var/lib/kubelet/pods/8f183a1c-3354-4637-891c-9bcbfb0daf67/volumes" Nov 03 21:14:18.896858 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.896844 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c97844f3-937c-4873-865a-c408d83b0890" path="/var/lib/kubelet/pods/c97844f3-937c-4873-865a-c408d83b0890/volumes" Nov 03 21:14:18.897600 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:18.897582 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc" path="/var/lib/kubelet/pods/fd3eeb4b-f1a2-4ba8-b094-f22ba1fc2afc/volumes" Nov 03 21:14:19.087371 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:19.087272 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-383e06ff-tzdkz"] Nov 03 21:14:19.090871 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:14:19.090840 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda20ce38b_39a2_4e14_92f9_ad37dd2ad124.slice/crio-c65ff9f9b6a4750b661d8642a3a04c069ed6d2bfab4a44b1480485052508d43a WatchSource:0}: Error finding container c65ff9f9b6a4750b661d8642a3a04c069ed6d2bfab4a44b1480485052508d43a: Status 404 returned error can't find the container with id c65ff9f9b6a4750b661d8642a3a04c069ed6d2bfab4a44b1480485052508d43a Nov 03 21:14:19.163931 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:19.163898 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adfa0d96-rqh4f"] Nov 03 21:14:19.166045 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:14:19.166019 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2e9589d_058b_47fb_a28d_9b92a855c9ca.slice/crio-aacb32aaf3bd4ac73b2479d8d1b941691f5162489c9bb15c1695d4cba07851e7 WatchSource:0}: Error finding container aacb32aaf3bd4ac73b2479d8d1b941691f5162489c9bb15c1695d4cba07851e7: Status 404 returned error can't find the container with id aacb32aaf3bd4ac73b2479d8d1b941691f5162489c9bb15c1695d4cba07851e7 Nov 03 21:14:19.751037 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:19.750995 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adfa0d96-rqh4f" event={"ID":"a2e9589d-058b-47fb-a28d-9b92a855c9ca","Type":"ContainerStarted","Data":"9fa7902258a909feffc1ebd2b8c620199f3b6eb406d744db68313d4ec51cf02c"} Nov 03 21:14:19.753225 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:19.751087 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adfa0d96-rqh4f" event={"ID":"a2e9589d-058b-47fb-a28d-9b92a855c9ca","Type":"ContainerStarted","Data":"aacb32aaf3bd4ac73b2479d8d1b941691f5162489c9bb15c1695d4cba07851e7"} Nov 03 21:14:19.754821 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:19.754777 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-383e06ff-tzdkz" event={"ID":"a20ce38b-39a2-4e14-92f9-ad37dd2ad124","Type":"ContainerStarted","Data":"bee3d344b6ddf6af03c750bb33e3fdb33209b01afbe44a1a21780d2bcd4ba8e3"} Nov 03 21:14:19.756462 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:19.754836 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-383e06ff-tzdkz" event={"ID":"a20ce38b-39a2-4e14-92f9-ad37dd2ad124","Type":"ContainerStarted","Data":"c65ff9f9b6a4750b661d8642a3a04c069ed6d2bfab4a44b1480485052508d43a"} Nov 03 21:14:19.870998 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:19.870936 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adfa0d96-rqh4f" podStartSLOduration=1.8708942020000001 podStartE2EDuration="1.870894202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:14:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:14:19.869365824 +0000 UTC m=+2095.094685640" watchObservedRunningTime="2026-11-03 21:14:19.870894202 +0000 UTC m=+2095.096213993" Nov 03 21:14:19.989400 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:19.989342 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-383e06ff-tzdkz" podStartSLOduration=1.9893233700000001 podStartE2EDuration="1.98932337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:14:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:14:19.98607062 +0000 UTC m=+2095.211390432" watchObservedRunningTime="2026-11-03 21:14:19.98932337 +0000 UTC m=+2095.214643164" Nov 03 21:14:20.439764 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:20.439720 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:14:20.763803 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:20.763712 4305 generic.go:356] "Generic (PLEG): container finished" podID="a20ce38b-39a2-4e14-92f9-ad37dd2ad124" containerID="bee3d344b6ddf6af03c750bb33e3fdb33209b01afbe44a1a21780d2bcd4ba8e3" exitCode=0 Nov 03 21:14:20.763803 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:20.763788 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-383e06ff-tzdkz" event={"ID":"a20ce38b-39a2-4e14-92f9-ad37dd2ad124","Type":"ContainerDied","Data":"bee3d344b6ddf6af03c750bb33e3fdb33209b01afbe44a1a21780d2bcd4ba8e3"} Nov 03 21:14:20.765650 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:20.765625 4305 generic.go:356] "Generic (PLEG): container finished" podID="a2e9589d-058b-47fb-a28d-9b92a855c9ca" containerID="9fa7902258a909feffc1ebd2b8c620199f3b6eb406d744db68313d4ec51cf02c" exitCode=0 Nov 03 21:14:20.765781 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:20.765716 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adfa0d96-rqh4f" event={"ID":"a2e9589d-058b-47fb-a28d-9b92a855c9ca","Type":"ContainerDied","Data":"9fa7902258a909feffc1ebd2b8c620199f3b6eb406d744db68313d4ec51cf02c"} Nov 03 21:14:21.937535 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:21.937506 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adfa0d96-rqh4f" Nov 03 21:14:21.968406 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:21.968380 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-383e06ff-tzdkz" Nov 03 21:14:21.998754 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:21.998716 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2e9589d-058b-47fb-a28d-9b92a855c9ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2e9589d-058b-47fb-a28d-9b92a855c9ca-maas-api-openshift-ca\") pod \"a2e9589d-058b-47fb-a28d-9b92a855c9ca\" (UID: \"a2e9589d-058b-47fb-a28d-9b92a855c9ca\") " Nov 03 21:14:21.998918 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:21.998780 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2e9589d-058b-47fb-a28d-9b92a855c9ca-kube-api-access-rsdfm\" (UniqueName: \"kubernetes.io/projected/a2e9589d-058b-47fb-a28d-9b92a855c9ca-kube-api-access-rsdfm\") pod \"a2e9589d-058b-47fb-a28d-9b92a855c9ca\" (UID: \"a2e9589d-058b-47fb-a28d-9b92a855c9ca\") " Nov 03 21:14:21.998918 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:21.998879 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2e9589d-058b-47fb-a28d-9b92a855c9ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2e9589d-058b-47fb-a28d-9b92a855c9ca-maas-api-xks-ca\") pod \"a2e9589d-058b-47fb-a28d-9b92a855c9ca\" (UID: \"a2e9589d-058b-47fb-a28d-9b92a855c9ca\") " Nov 03 21:14:21.999097 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:21.999075 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2e9589d-058b-47fb-a28d-9b92a855c9ca-maas-api-openshift-ca" pod "a2e9589d-058b-47fb-a28d-9b92a855c9ca" (UID: "a2e9589d-058b-47fb-a28d-9b92a855c9ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:14:22.000779 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:22.000734 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2e9589d-058b-47fb-a28d-9b92a855c9ca-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:22.001386 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:22.001369 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2e9589d-058b-47fb-a28d-9b92a855c9ca-maas-api-xks-ca" pod "a2e9589d-058b-47fb-a28d-9b92a855c9ca" (UID: "a2e9589d-058b-47fb-a28d-9b92a855c9ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:14:22.001739 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:22.001716 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2e9589d-058b-47fb-a28d-9b92a855c9ca-kube-api-access-rsdfm" pod "a2e9589d-058b-47fb-a28d-9b92a855c9ca" (UID: "a2e9589d-058b-47fb-a28d-9b92a855c9ca"). InnerVolumeSpecName "kube-api-access-rsdfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:14:22.101656 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:22.101568 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a20ce38b-39a2-4e14-92f9-ad37dd2ad124-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a20ce38b-39a2-4e14-92f9-ad37dd2ad124-maas-api-xks-ca\") pod \"a20ce38b-39a2-4e14-92f9-ad37dd2ad124\" (UID: \"a20ce38b-39a2-4e14-92f9-ad37dd2ad124\") " Nov 03 21:14:22.101821 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:22.101658 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a20ce38b-39a2-4e14-92f9-ad37dd2ad124-kube-api-access-vvl9f\" (UniqueName: \"kubernetes.io/projected/a20ce38b-39a2-4e14-92f9-ad37dd2ad124-kube-api-access-vvl9f\") pod \"a20ce38b-39a2-4e14-92f9-ad37dd2ad124\" (UID: \"a20ce38b-39a2-4e14-92f9-ad37dd2ad124\") " Nov 03 21:14:22.101821 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:22.101757 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a20ce38b-39a2-4e14-92f9-ad37dd2ad124-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a20ce38b-39a2-4e14-92f9-ad37dd2ad124-maas-api-openshift-ca\") pod \"a20ce38b-39a2-4e14-92f9-ad37dd2ad124\" (UID: \"a20ce38b-39a2-4e14-92f9-ad37dd2ad124\") " Nov 03 21:14:22.102044 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:22.102020 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2e9589d-058b-47fb-a28d-9b92a855c9ca-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:22.102044 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:22.102044 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsdfm\" (UniqueName: \"kubernetes.io/projected/a2e9589d-058b-47fb-a28d-9b92a855c9ca-kube-api-access-rsdfm\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:22.102202 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:22.102151 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a20ce38b-39a2-4e14-92f9-ad37dd2ad124-maas-api-openshift-ca" pod "a20ce38b-39a2-4e14-92f9-ad37dd2ad124" (UID: "a20ce38b-39a2-4e14-92f9-ad37dd2ad124"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:14:22.103958 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:22.103930 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a20ce38b-39a2-4e14-92f9-ad37dd2ad124-maas-api-xks-ca" pod "a20ce38b-39a2-4e14-92f9-ad37dd2ad124" (UID: "a20ce38b-39a2-4e14-92f9-ad37dd2ad124"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:14:22.104054 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:22.103992 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a20ce38b-39a2-4e14-92f9-ad37dd2ad124-kube-api-access-vvl9f" pod "a20ce38b-39a2-4e14-92f9-ad37dd2ad124" (UID: "a20ce38b-39a2-4e14-92f9-ad37dd2ad124"). InnerVolumeSpecName "kube-api-access-vvl9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:14:22.203113 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:22.203077 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a20ce38b-39a2-4e14-92f9-ad37dd2ad124-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:22.203113 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:22.203111 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvl9f\" (UniqueName: \"kubernetes.io/projected/a20ce38b-39a2-4e14-92f9-ad37dd2ad124-kube-api-access-vvl9f\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:22.203325 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:22.203124 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a20ce38b-39a2-4e14-92f9-ad37dd2ad124-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:22.784929 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:22.784903 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adfa0d96-rqh4f" Nov 03 21:14:22.785126 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:22.784942 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adfa0d96-rqh4f" event={"ID":"a2e9589d-058b-47fb-a28d-9b92a855c9ca","Type":"ContainerDied","Data":"aacb32aaf3bd4ac73b2479d8d1b941691f5162489c9bb15c1695d4cba07851e7"} Nov 03 21:14:22.785126 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:22.784999 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aacb32aaf3bd4ac73b2479d8d1b941691f5162489c9bb15c1695d4cba07851e7" Nov 03 21:14:22.786574 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:22.786548 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-383e06ff-tzdkz" event={"ID":"a20ce38b-39a2-4e14-92f9-ad37dd2ad124","Type":"ContainerDied","Data":"c65ff9f9b6a4750b661d8642a3a04c069ed6d2bfab4a44b1480485052508d43a"} Nov 03 21:14:22.786574 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:22.786571 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-383e06ff-tzdkz" Nov 03 21:14:22.786767 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:22.786589 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c65ff9f9b6a4750b661d8642a3a04c069ed6d2bfab4a44b1480485052508d43a" Nov 03 21:14:23.257151 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:23.257106 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-862138-65fb688c66-fx4kk" podUID="6c3bdfd9-c058-49d6-9179-4a53b523216f" containerName="payload-processing" containerID="cri-o://1b30d77ad6e760a8dec5065402b6f262ec0231531ac296a326ee1ad0abfa0e98" gracePeriod=25 Nov 03 21:14:23.388870 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:23.388824 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-862138-6c8d9c5567-7nvj6" podUID="3e3a291b-d5be-4d9c-8b8a-0b97dba864e2" containerName="payload-pre-processing" containerID="cri-o://fd4421526c4ecde741d5b2716623486237c2c7688cb46b1c2f684a68ed674acd" gracePeriod=25 Nov 03 21:14:25.364555 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:25.364510 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-862138-65fb688c66-fx4kk" podUID="6c3bdfd9-c058-49d6-9179-4a53b523216f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:14:25.566092 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:25.566041 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e22abf-hncdr"] Nov 03 21:14:25.615523 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:25.615444 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-e22abf-hncdr"] Nov 03 21:14:26.895278 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:26.895239 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da6b7530-9a01-4bfb-bf70-dff574ef1f0f" path="/var/lib/kubelet/pods/da6b7530-9a01-4bfb-bf70-dff574ef1f0f/volumes" Nov 03 21:14:28.173278 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:28.173240 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-12a423-846d4b6867-grwcg" podUID="7b0135ee-b547-46e8-9953-6a46cb61f2c4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:14:28.173774 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:28.173357 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-12a423-559b96f96d-nmf2b" podUID="3e6b8d49-0db7-4bbc-9b3e-1105d2bdbc78" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:14:31.862545 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:31.862512 4305 scope.go:122] "RemoveContainer" containerID="38897c2dd34e6fa54c70eb02bea42b5b623cfcdd4bb07c17be52d76a440fc8c7" Nov 03 21:14:31.873066 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:31.873042 4305 scope.go:122] "RemoveContainer" containerID="8694ce683875e96acaa705b446df26bf253f5aec162f7500538b0c74f0fffddb" Nov 03 21:14:31.883064 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:31.883043 4305 scope.go:122] "RemoveContainer" containerID="0031641bfc57cf20bfe768a2d6924d674e775b54841e550dba97573cee0e0c95" Nov 03 21:14:31.892406 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:31.892386 4305 scope.go:122] "RemoveContainer" containerID="8c45ca20ee60a442b554da2588707fbf2c60bc1b19a26aab9296279b1cf37ea5" Nov 03 21:14:31.954078 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:31.954040 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-862138-k4bq6"] Nov 03 21:14:32.000491 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:32.000466 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-862138-k4bq6"] Nov 03 21:14:32.906751 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:32.906332 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a568044d-1a39-4916-86ea-fadfbcb6ddcd" path="/var/lib/kubelet/pods/a568044d-1a39-4916-86ea-fadfbcb6ddcd/volumes" Nov 03 21:14:33.074480 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:33.073477 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-12a423-559b96f96d-nmf2b" Nov 03 21:14:33.168970 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:33.168928 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-12a423-846d4b6867-grwcg" Nov 03 21:14:33.234823 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:33.234783 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e6b8d49-0db7-4bbc-9b3e-1105d2bdbc78-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e6b8d49-0db7-4bbc-9b3e-1105d2bdbc78-plugins-config-volume\") pod \"3e6b8d49-0db7-4bbc-9b3e-1105d2bdbc78\" (UID: \"3e6b8d49-0db7-4bbc-9b3e-1105d2bdbc78\") " Nov 03 21:14:33.235036 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:33.234914 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e6b8d49-0db7-4bbc-9b3e-1105d2bdbc78-kube-api-access-4jtr2\" (UniqueName: \"kubernetes.io/projected/3e6b8d49-0db7-4bbc-9b3e-1105d2bdbc78-kube-api-access-4jtr2\") pod \"3e6b8d49-0db7-4bbc-9b3e-1105d2bdbc78\" (UID: \"3e6b8d49-0db7-4bbc-9b3e-1105d2bdbc78\") " Nov 03 21:14:33.235178 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:33.235153 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e6b8d49-0db7-4bbc-9b3e-1105d2bdbc78-plugins-config-volume" pod "3e6b8d49-0db7-4bbc-9b3e-1105d2bdbc78" (UID: "3e6b8d49-0db7-4bbc-9b3e-1105d2bdbc78"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:14:33.235339 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:33.235322 4305 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e6b8d49-0db7-4bbc-9b3e-1105d2bdbc78-plugins-config-volume\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:33.237246 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:33.237225 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e6b8d49-0db7-4bbc-9b3e-1105d2bdbc78-kube-api-access-4jtr2" pod "3e6b8d49-0db7-4bbc-9b3e-1105d2bdbc78" (UID: "3e6b8d49-0db7-4bbc-9b3e-1105d2bdbc78"). InnerVolumeSpecName "kube-api-access-4jtr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:14:33.250850 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:33.250821 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-12a423-2hvxd"] Nov 03 21:14:33.258102 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:33.258067 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-12a423-2hvxd"] Nov 03 21:14:33.336034 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:33.335947 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b0135ee-b547-46e8-9953-6a46cb61f2c4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b0135ee-b547-46e8-9953-6a46cb61f2c4-plugins-config-volume\") pod \"7b0135ee-b547-46e8-9953-6a46cb61f2c4\" (UID: \"7b0135ee-b547-46e8-9953-6a46cb61f2c4\") " Nov 03 21:14:33.336196 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:33.336123 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b0135ee-b547-46e8-9953-6a46cb61f2c4-kube-api-access-v4h57\" (UniqueName: \"kubernetes.io/projected/7b0135ee-b547-46e8-9953-6a46cb61f2c4-kube-api-access-v4h57\") pod \"7b0135ee-b547-46e8-9953-6a46cb61f2c4\" (UID: \"7b0135ee-b547-46e8-9953-6a46cb61f2c4\") " Nov 03 21:14:33.336409 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:33.336387 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b0135ee-b547-46e8-9953-6a46cb61f2c4-plugins-config-volume" pod "7b0135ee-b547-46e8-9953-6a46cb61f2c4" (UID: "7b0135ee-b547-46e8-9953-6a46cb61f2c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:14:33.336543 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:33.336530 4305 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b0135ee-b547-46e8-9953-6a46cb61f2c4-plugins-config-volume\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:33.336583 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:33.336546 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jtr2\" (UniqueName: \"kubernetes.io/projected/3e6b8d49-0db7-4bbc-9b3e-1105d2bdbc78-kube-api-access-4jtr2\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:33.338548 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:33.338524 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b0135ee-b547-46e8-9953-6a46cb61f2c4-kube-api-access-v4h57" pod "7b0135ee-b547-46e8-9953-6a46cb61f2c4" (UID: "7b0135ee-b547-46e8-9953-6a46cb61f2c4"). InnerVolumeSpecName "kube-api-access-v4h57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:14:33.345783 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:33.345751 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-862138-6c8d9c5567-7nvj6" podUID="3e3a291b-d5be-4d9c-8b8a-0b97dba864e2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:14:33.438017 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:33.437979 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4h57\" (UniqueName: \"kubernetes.io/projected/7b0135ee-b547-46e8-9953-6a46cb61f2c4-kube-api-access-v4h57\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:33.870890 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:33.870852 4305 generic.go:356] "Generic (PLEG): container finished" podID="7b0135ee-b547-46e8-9953-6a46cb61f2c4" containerID="e69e1c1980274f4b77b456df4c88e10de37015cb0a6d99dfa49f4a8f433dfd4c" exitCode=0 Nov 03 21:14:33.871098 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:33.870923 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-12a423-846d4b6867-grwcg" Nov 03 21:14:33.871098 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:33.870934 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-12a423-846d4b6867-grwcg" event={"ID":"7b0135ee-b547-46e8-9953-6a46cb61f2c4","Type":"ContainerDied","Data":"e69e1c1980274f4b77b456df4c88e10de37015cb0a6d99dfa49f4a8f433dfd4c"} Nov 03 21:14:33.871098 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:33.871024 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-12a423-846d4b6867-grwcg" event={"ID":"7b0135ee-b547-46e8-9953-6a46cb61f2c4","Type":"ContainerDied","Data":"5b57d66efcbd3ed9ffff0dcb6c4b92d686e0041ced2b2611ee08b4defc4b3c75"} Nov 03 21:14:33.871098 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:33.871054 4305 scope.go:122] "RemoveContainer" containerID="e69e1c1980274f4b77b456df4c88e10de37015cb0a6d99dfa49f4a8f433dfd4c" Nov 03 21:14:33.872342 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:33.872321 4305 generic.go:356] "Generic (PLEG): container finished" podID="3e6b8d49-0db7-4bbc-9b3e-1105d2bdbc78" containerID="3f479753f615643b9fe749dbdaeb5ddde4eace1984c7faab8edcedfef05c9ea3" exitCode=0 Nov 03 21:14:33.872457 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:33.872378 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-12a423-559b96f96d-nmf2b" Nov 03 21:14:33.872457 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:33.872410 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-12a423-559b96f96d-nmf2b" event={"ID":"3e6b8d49-0db7-4bbc-9b3e-1105d2bdbc78","Type":"ContainerDied","Data":"3f479753f615643b9fe749dbdaeb5ddde4eace1984c7faab8edcedfef05c9ea3"} Nov 03 21:14:33.872596 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:33.872487 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-12a423-559b96f96d-nmf2b" event={"ID":"3e6b8d49-0db7-4bbc-9b3e-1105d2bdbc78","Type":"ContainerDied","Data":"d64cd7e6e5dedffae22c1db7aab85e69beb880716678a18ff99d8a1a28e9d10d"} Nov 03 21:14:33.881226 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:33.881118 4305 scope.go:122] "RemoveContainer" containerID="e69e1c1980274f4b77b456df4c88e10de37015cb0a6d99dfa49f4a8f433dfd4c" Nov 03 21:14:33.881543 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:14:33.881523 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e69e1c1980274f4b77b456df4c88e10de37015cb0a6d99dfa49f4a8f433dfd4c\": container with ID starting with e69e1c1980274f4b77b456df4c88e10de37015cb0a6d99dfa49f4a8f433dfd4c not found: ID does not exist" containerID="e69e1c1980274f4b77b456df4c88e10de37015cb0a6d99dfa49f4a8f433dfd4c" Nov 03 21:14:33.881602 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:33.881552 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e69e1c1980274f4b77b456df4c88e10de37015cb0a6d99dfa49f4a8f433dfd4c"} err="failed to get container status \"e69e1c1980274f4b77b456df4c88e10de37015cb0a6d99dfa49f4a8f433dfd4c\": rpc error: code = NotFound desc = could not find container \"e69e1c1980274f4b77b456df4c88e10de37015cb0a6d99dfa49f4a8f433dfd4c\": container with ID starting with e69e1c1980274f4b77b456df4c88e10de37015cb0a6d99dfa49f4a8f433dfd4c not found: ID does not exist" Nov 03 21:14:33.881602 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:33.881571 4305 scope.go:122] "RemoveContainer" containerID="3f479753f615643b9fe749dbdaeb5ddde4eace1984c7faab8edcedfef05c9ea3" Nov 03 21:14:33.891556 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:33.891529 4305 scope.go:122] "RemoveContainer" containerID="3f479753f615643b9fe749dbdaeb5ddde4eace1984c7faab8edcedfef05c9ea3" Nov 03 21:14:33.891877 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:14:33.891854 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f479753f615643b9fe749dbdaeb5ddde4eace1984c7faab8edcedfef05c9ea3\": container with ID starting with 3f479753f615643b9fe749dbdaeb5ddde4eace1984c7faab8edcedfef05c9ea3 not found: ID does not exist" containerID="3f479753f615643b9fe749dbdaeb5ddde4eace1984c7faab8edcedfef05c9ea3" Nov 03 21:14:33.891969 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:33.891886 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f479753f615643b9fe749dbdaeb5ddde4eace1984c7faab8edcedfef05c9ea3"} err="failed to get container status \"3f479753f615643b9fe749dbdaeb5ddde4eace1984c7faab8edcedfef05c9ea3\": rpc error: code = NotFound desc = could not find container \"3f479753f615643b9fe749dbdaeb5ddde4eace1984c7faab8edcedfef05c9ea3\": container with ID starting with 3f479753f615643b9fe749dbdaeb5ddde4eace1984c7faab8edcedfef05c9ea3 not found: ID does not exist" Nov 03 21:14:34.035989 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:34.035955 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-12a423-846d4b6867-grwcg"] Nov 03 21:14:34.040141 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:34.040116 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-12a423-846d4b6867-grwcg"] Nov 03 21:14:34.152603 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:34.152570 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-12a423-559b96f96d-nmf2b"] Nov 03 21:14:34.155328 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:34.155304 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-12a423-559b96f96d-nmf2b"] Nov 03 21:14:34.545558 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:34.545426 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v"] Nov 03 21:14:34.545724 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:34.545672 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v" podUID="b24ff37e-1709-4f11-b0e8-9302b9f3d9ad" containerName="istio-proxy" containerID="cri-o://6e1aae7afc7ed5b987a6f7b40e8fb7e76b95e6fd2665fa3f61350f63239d6d27" gracePeriod=30 Nov 03 21:14:34.894831 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:34.894801 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e6b8d49-0db7-4bbc-9b3e-1105d2bdbc78" path="/var/lib/kubelet/pods/3e6b8d49-0db7-4bbc-9b3e-1105d2bdbc78/volumes" Nov 03 21:14:34.895198 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:34.895185 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b0135ee-b547-46e8-9953-6a46cb61f2c4" path="/var/lib/kubelet/pods/7b0135ee-b547-46e8-9953-6a46cb61f2c4/volumes" Nov 03 21:14:34.895619 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:34.895606 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b488427-d78e-41c3-848d-b5cf765a1e83" path="/var/lib/kubelet/pods/9b488427-d78e-41c3-848d-b5cf765a1e83/volumes" Nov 03 21:14:35.364502 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:35.364383 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-862138-65fb688c66-fx4kk" podUID="6c3bdfd9-c058-49d6-9179-4a53b523216f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:14:38.522262 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:38.522168 4305 patch_prober.go:29] interesting pod/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:14:38.522730 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:38.522255 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v" podUID="b24ff37e-1709-4f11-b0e8-9302b9f3d9ad" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:14:38.916446 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:38.916388 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn"] Nov 03 21:14:38.916686 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:38.916639 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn" podUID="2fea2251-d380-4c79-a754-1a3ec7bdf829" containerName="istio-proxy" containerID="cri-o://caa40f2d41760bebe3ffe2434ec0064e45f551b4431a64c9fd2351ec5f854170" gracePeriod=30 Nov 03 21:14:39.804950 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:39.804922 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v" Nov 03 21:14:39.904997 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:39.904964 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b24ff37e-1709-4f11-b0e8-9302b9f3d9ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24ff37e-1709-4f11-b0e8-9302b9f3d9ad-workload-socket\") pod \"b24ff37e-1709-4f11-b0e8-9302b9f3d9ad\" (UID: \"b24ff37e-1709-4f11-b0e8-9302b9f3d9ad\") " Nov 03 21:14:39.904997 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:39.905002 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b24ff37e-1709-4f11-b0e8-9302b9f3d9ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24ff37e-1709-4f11-b0e8-9302b9f3d9ad-credential-socket\") pod \"b24ff37e-1709-4f11-b0e8-9302b9f3d9ad\" (UID: \"b24ff37e-1709-4f11-b0e8-9302b9f3d9ad\") " Nov 03 21:14:39.905262 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:39.905041 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b24ff37e-1709-4f11-b0e8-9302b9f3d9ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b24ff37e-1709-4f11-b0e8-9302b9f3d9ad-workload-certs\") pod \"b24ff37e-1709-4f11-b0e8-9302b9f3d9ad\" (UID: \"b24ff37e-1709-4f11-b0e8-9302b9f3d9ad\") " Nov 03 21:14:39.905262 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:39.905073 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b24ff37e-1709-4f11-b0e8-9302b9f3d9ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b24ff37e-1709-4f11-b0e8-9302b9f3d9ad-istiod-ca-cert\") pod \"b24ff37e-1709-4f11-b0e8-9302b9f3d9ad\" (UID: \"b24ff37e-1709-4f11-b0e8-9302b9f3d9ad\") " Nov 03 21:14:39.905262 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:39.905137 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b24ff37e-1709-4f11-b0e8-9302b9f3d9ad-kube-api-access-c86md\" (UniqueName: \"kubernetes.io/projected/b24ff37e-1709-4f11-b0e8-9302b9f3d9ad-kube-api-access-c86md\") pod \"b24ff37e-1709-4f11-b0e8-9302b9f3d9ad\" (UID: \"b24ff37e-1709-4f11-b0e8-9302b9f3d9ad\") " Nov 03 21:14:39.905262 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:39.905174 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b24ff37e-1709-4f11-b0e8-9302b9f3d9ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b24ff37e-1709-4f11-b0e8-9302b9f3d9ad-istio-envoy\") pod \"b24ff37e-1709-4f11-b0e8-9302b9f3d9ad\" (UID: \"b24ff37e-1709-4f11-b0e8-9302b9f3d9ad\") " Nov 03 21:14:39.905262 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:39.905194 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b24ff37e-1709-4f11-b0e8-9302b9f3d9ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b24ff37e-1709-4f11-b0e8-9302b9f3d9ad-istio-podinfo\") pod \"b24ff37e-1709-4f11-b0e8-9302b9f3d9ad\" (UID: \"b24ff37e-1709-4f11-b0e8-9302b9f3d9ad\") " Nov 03 21:14:39.905262 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:39.905221 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b24ff37e-1709-4f11-b0e8-9302b9f3d9ad-istio-token\" (UniqueName: \"kubernetes.io/projected/b24ff37e-1709-4f11-b0e8-9302b9f3d9ad-istio-token\") pod \"b24ff37e-1709-4f11-b0e8-9302b9f3d9ad\" (UID: \"b24ff37e-1709-4f11-b0e8-9302b9f3d9ad\") " Nov 03 21:14:39.905262 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:39.905252 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b24ff37e-1709-4f11-b0e8-9302b9f3d9ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b24ff37e-1709-4f11-b0e8-9302b9f3d9ad-istio-data\") pod \"b24ff37e-1709-4f11-b0e8-9302b9f3d9ad\" (UID: \"b24ff37e-1709-4f11-b0e8-9302b9f3d9ad\") " Nov 03 21:14:39.905632 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:39.905326 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b24ff37e-1709-4f11-b0e8-9302b9f3d9ad-workload-socket" pod "b24ff37e-1709-4f11-b0e8-9302b9f3d9ad" (UID: "b24ff37e-1709-4f11-b0e8-9302b9f3d9ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:39.905632 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:39.905353 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b24ff37e-1709-4f11-b0e8-9302b9f3d9ad-credential-socket" pod "b24ff37e-1709-4f11-b0e8-9302b9f3d9ad" (UID: "b24ff37e-1709-4f11-b0e8-9302b9f3d9ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:39.905632 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:39.905429 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b24ff37e-1709-4f11-b0e8-9302b9f3d9ad-workload-certs" pod "b24ff37e-1709-4f11-b0e8-9302b9f3d9ad" (UID: "b24ff37e-1709-4f11-b0e8-9302b9f3d9ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:39.905632 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:39.905563 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b24ff37e-1709-4f11-b0e8-9302b9f3d9ad-istio-data" pod "b24ff37e-1709-4f11-b0e8-9302b9f3d9ad" (UID: "b24ff37e-1709-4f11-b0e8-9302b9f3d9ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:39.905632 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:39.905563 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b24ff37e-1709-4f11-b0e8-9302b9f3d9ad-istiod-ca-cert" pod "b24ff37e-1709-4f11-b0e8-9302b9f3d9ad" (UID: "b24ff37e-1709-4f11-b0e8-9302b9f3d9ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:14:39.905941 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:39.905780 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24ff37e-1709-4f11-b0e8-9302b9f3d9ad-workload-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:39.905941 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:39.905802 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24ff37e-1709-4f11-b0e8-9302b9f3d9ad-credential-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:39.905941 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:39.905818 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b24ff37e-1709-4f11-b0e8-9302b9f3d9ad-workload-certs\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:39.905941 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:39.905831 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b24ff37e-1709-4f11-b0e8-9302b9f3d9ad-istiod-ca-cert\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:39.905941 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:39.905843 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b24ff37e-1709-4f11-b0e8-9302b9f3d9ad-istio-data\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:39.907699 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:39.907674 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b24ff37e-1709-4f11-b0e8-9302b9f3d9ad-kube-api-access-c86md" pod "b24ff37e-1709-4f11-b0e8-9302b9f3d9ad" (UID: "b24ff37e-1709-4f11-b0e8-9302b9f3d9ad"). InnerVolumeSpecName "kube-api-access-c86md". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:14:39.907807 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:39.907712 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b24ff37e-1709-4f11-b0e8-9302b9f3d9ad-istio-token" pod "b24ff37e-1709-4f11-b0e8-9302b9f3d9ad" (UID: "b24ff37e-1709-4f11-b0e8-9302b9f3d9ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:14:39.907807 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:39.907794 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b24ff37e-1709-4f11-b0e8-9302b9f3d9ad-istio-envoy" pod "b24ff37e-1709-4f11-b0e8-9302b9f3d9ad" (UID: "b24ff37e-1709-4f11-b0e8-9302b9f3d9ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:39.907888 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:39.907860 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b24ff37e-1709-4f11-b0e8-9302b9f3d9ad-istio-podinfo" pod "b24ff37e-1709-4f11-b0e8-9302b9f3d9ad" (UID: "b24ff37e-1709-4f11-b0e8-9302b9f3d9ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:14:39.917448 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:39.917405 4305 generic.go:356] "Generic (PLEG): container finished" podID="b24ff37e-1709-4f11-b0e8-9302b9f3d9ad" containerID="6e1aae7afc7ed5b987a6f7b40e8fb7e76b95e6fd2665fa3f61350f63239d6d27" exitCode=0 Nov 03 21:14:39.917589 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:39.917538 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v" Nov 03 21:14:39.917589 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:39.917565 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v" event={"ID":"b24ff37e-1709-4f11-b0e8-9302b9f3d9ad","Type":"ContainerDied","Data":"6e1aae7afc7ed5b987a6f7b40e8fb7e76b95e6fd2665fa3f61350f63239d6d27"} Nov 03 21:14:39.917703 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:39.917618 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v" event={"ID":"b24ff37e-1709-4f11-b0e8-9302b9f3d9ad","Type":"ContainerDied","Data":"57dfd54234ef7a9ff5f0ffc62dec91d4d88c544721ed16c18400c0bc17fc7b78"} Nov 03 21:14:39.917703 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:39.917640 4305 scope.go:122] "RemoveContainer" containerID="6e1aae7afc7ed5b987a6f7b40e8fb7e76b95e6fd2665fa3f61350f63239d6d27" Nov 03 21:14:39.933984 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:39.933956 4305 scope.go:122] "RemoveContainer" containerID="6e1aae7afc7ed5b987a6f7b40e8fb7e76b95e6fd2665fa3f61350f63239d6d27" Nov 03 21:14:39.934370 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:14:39.934348 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e1aae7afc7ed5b987a6f7b40e8fb7e76b95e6fd2665fa3f61350f63239d6d27\": container with ID starting with 6e1aae7afc7ed5b987a6f7b40e8fb7e76b95e6fd2665fa3f61350f63239d6d27 not found: ID does not exist" containerID="6e1aae7afc7ed5b987a6f7b40e8fb7e76b95e6fd2665fa3f61350f63239d6d27" Nov 03 21:14:39.934651 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:39.934380 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e1aae7afc7ed5b987a6f7b40e8fb7e76b95e6fd2665fa3f61350f63239d6d27"} err="failed to get container status \"6e1aae7afc7ed5b987a6f7b40e8fb7e76b95e6fd2665fa3f61350f63239d6d27\": rpc error: code = NotFound desc = could not find container \"6e1aae7afc7ed5b987a6f7b40e8fb7e76b95e6fd2665fa3f61350f63239d6d27\": container with ID starting with 6e1aae7afc7ed5b987a6f7b40e8fb7e76b95e6fd2665fa3f61350f63239d6d27 not found: ID does not exist" Nov 03 21:14:40.006375 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:40.006331 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c86md\" (UniqueName: \"kubernetes.io/projected/b24ff37e-1709-4f11-b0e8-9302b9f3d9ad-kube-api-access-c86md\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:40.006375 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:40.006365 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b24ff37e-1709-4f11-b0e8-9302b9f3d9ad-istio-envoy\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:40.006375 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:40.006379 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b24ff37e-1709-4f11-b0e8-9302b9f3d9ad-istio-podinfo\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:40.006640 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:40.006390 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b24ff37e-1709-4f11-b0e8-9302b9f3d9ad-istio-token\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:40.098422 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:40.098387 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v"] Nov 03 21:14:40.100758 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:40.100734 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-e22abf-openshift-default-5f77dbbb7-qq58v"] Nov 03 21:14:40.893927 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:40.893898 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b24ff37e-1709-4f11-b0e8-9302b9f3d9ad" path="/var/lib/kubelet/pods/b24ff37e-1709-4f11-b0e8-9302b9f3d9ad/volumes" Nov 03 21:14:40.926101 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:40.926074 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr"] Nov 03 21:14:40.926304 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:40.926280 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr" podUID="fbb09c2d-a432-4507-a7d2-9e401f0e96bb" containerName="istio-proxy" containerID="cri-o://c3207900e035579b7be53a7527c6e070412de23241a06f4490a143d042b544e6" gracePeriod=30 Nov 03 21:14:41.269558 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:41.269477 4305 patch_prober.go:29] interesting pod/e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:14:41.269725 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:41.269547 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr" podUID="fbb09c2d-a432-4507-a7d2-9e401f0e96bb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:14:42.808632 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:42.808593 4305 patch_prober.go:29] interesting pod/e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:14:42.809041 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:42.808651 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn" podUID="2fea2251-d380-4c79-a754-1a3ec7bdf829" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:14:43.346337 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:43.346302 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-862138-6c8d9c5567-7nvj6" podUID="3e3a291b-d5be-4d9c-8b8a-0b97dba864e2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:14:43.516686 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:43.516624 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-862138-65fb688c66-fx4kk" Nov 03 21:14:43.537290 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:43.537261 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c3bdfd9-c058-49d6-9179-4a53b523216f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c3bdfd9-c058-49d6-9179-4a53b523216f-plugins-config-volume\") pod \"6c3bdfd9-c058-49d6-9179-4a53b523216f\" (UID: \"6c3bdfd9-c058-49d6-9179-4a53b523216f\") " Nov 03 21:14:43.537511 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:43.537313 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c3bdfd9-c058-49d6-9179-4a53b523216f-kube-api-access-z4rc5\" (UniqueName: \"kubernetes.io/projected/6c3bdfd9-c058-49d6-9179-4a53b523216f-kube-api-access-z4rc5\") pod \"6c3bdfd9-c058-49d6-9179-4a53b523216f\" (UID: \"6c3bdfd9-c058-49d6-9179-4a53b523216f\") " Nov 03 21:14:43.537745 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:43.537717 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c3bdfd9-c058-49d6-9179-4a53b523216f-plugins-config-volume" pod "6c3bdfd9-c058-49d6-9179-4a53b523216f" (UID: "6c3bdfd9-c058-49d6-9179-4a53b523216f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:14:43.539883 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:43.539845 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c3bdfd9-c058-49d6-9179-4a53b523216f-kube-api-access-z4rc5" pod "6c3bdfd9-c058-49d6-9179-4a53b523216f" (UID: "6c3bdfd9-c058-49d6-9179-4a53b523216f"). InnerVolumeSpecName "kube-api-access-z4rc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:14:43.638937 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:43.638902 4305 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c3bdfd9-c058-49d6-9179-4a53b523216f-plugins-config-volume\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:43.639120 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:43.638966 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4rc5\" (UniqueName: \"kubernetes.io/projected/6c3bdfd9-c058-49d6-9179-4a53b523216f-kube-api-access-z4rc5\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:43.639120 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:43.639051 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-862138-6c8d9c5567-7nvj6" Nov 03 21:14:43.740129 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:43.740092 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e3a291b-d5be-4d9c-8b8a-0b97dba864e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e3a291b-d5be-4d9c-8b8a-0b97dba864e2-plugins-config-volume\") pod \"3e3a291b-d5be-4d9c-8b8a-0b97dba864e2\" (UID: \"3e3a291b-d5be-4d9c-8b8a-0b97dba864e2\") " Nov 03 21:14:43.740346 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:43.740160 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e3a291b-d5be-4d9c-8b8a-0b97dba864e2-kube-api-access-ldqvn\" (UniqueName: \"kubernetes.io/projected/3e3a291b-d5be-4d9c-8b8a-0b97dba864e2-kube-api-access-ldqvn\") pod \"3e3a291b-d5be-4d9c-8b8a-0b97dba864e2\" (UID: \"3e3a291b-d5be-4d9c-8b8a-0b97dba864e2\") " Nov 03 21:14:43.740535 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:43.740506 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e3a291b-d5be-4d9c-8b8a-0b97dba864e2-plugins-config-volume" pod "3e3a291b-d5be-4d9c-8b8a-0b97dba864e2" (UID: "3e3a291b-d5be-4d9c-8b8a-0b97dba864e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:14:43.742517 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:43.742480 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e3a291b-d5be-4d9c-8b8a-0b97dba864e2-kube-api-access-ldqvn" pod "3e3a291b-d5be-4d9c-8b8a-0b97dba864e2" (UID: "3e3a291b-d5be-4d9c-8b8a-0b97dba864e2"). InnerVolumeSpecName "kube-api-access-ldqvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:14:43.841002 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:43.840913 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldqvn\" (UniqueName: \"kubernetes.io/projected/3e3a291b-d5be-4d9c-8b8a-0b97dba864e2-kube-api-access-ldqvn\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:43.841002 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:43.840948 4305 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e3a291b-d5be-4d9c-8b8a-0b97dba864e2-plugins-config-volume\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:43.948083 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:43.948048 4305 generic.go:356] "Generic (PLEG): container finished" podID="3e3a291b-d5be-4d9c-8b8a-0b97dba864e2" containerID="fd4421526c4ecde741d5b2716623486237c2c7688cb46b1c2f684a68ed674acd" exitCode=0 Nov 03 21:14:43.948235 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:43.948153 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-862138-6c8d9c5567-7nvj6" Nov 03 21:14:43.948235 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:43.948178 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-862138-6c8d9c5567-7nvj6" event={"ID":"3e3a291b-d5be-4d9c-8b8a-0b97dba864e2","Type":"ContainerDied","Data":"fd4421526c4ecde741d5b2716623486237c2c7688cb46b1c2f684a68ed674acd"} Nov 03 21:14:43.948235 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:43.948231 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-862138-6c8d9c5567-7nvj6" event={"ID":"3e3a291b-d5be-4d9c-8b8a-0b97dba864e2","Type":"ContainerDied","Data":"73704d0429ed3c325c85b49896bbaa436f0f1742d97594c04544b19d6a420269"} Nov 03 21:14:43.948379 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:43.948259 4305 scope.go:122] "RemoveContainer" containerID="fd4421526c4ecde741d5b2716623486237c2c7688cb46b1c2f684a68ed674acd" Nov 03 21:14:43.950229 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:43.950200 4305 generic.go:356] "Generic (PLEG): container finished" podID="6c3bdfd9-c058-49d6-9179-4a53b523216f" containerID="1b30d77ad6e760a8dec5065402b6f262ec0231531ac296a326ee1ad0abfa0e98" exitCode=0 Nov 03 21:14:43.950346 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:43.950261 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-862138-65fb688c66-fx4kk" Nov 03 21:14:43.950346 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:43.950277 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-862138-65fb688c66-fx4kk" event={"ID":"6c3bdfd9-c058-49d6-9179-4a53b523216f","Type":"ContainerDied","Data":"1b30d77ad6e760a8dec5065402b6f262ec0231531ac296a326ee1ad0abfa0e98"} Nov 03 21:14:43.950346 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:43.950336 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-862138-65fb688c66-fx4kk" event={"ID":"6c3bdfd9-c058-49d6-9179-4a53b523216f","Type":"ContainerDied","Data":"7f7dd5f3705a424f45e9ada4c5279052c881f0fcb58ea3b202d5042e7fcf341f"} Nov 03 21:14:44.038255 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:44.038227 4305 scope.go:122] "RemoveContainer" containerID="fd4421526c4ecde741d5b2716623486237c2c7688cb46b1c2f684a68ed674acd" Nov 03 21:14:44.038704 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:14:44.038679 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd4421526c4ecde741d5b2716623486237c2c7688cb46b1c2f684a68ed674acd\": container with ID starting with fd4421526c4ecde741d5b2716623486237c2c7688cb46b1c2f684a68ed674acd not found: ID does not exist" containerID="fd4421526c4ecde741d5b2716623486237c2c7688cb46b1c2f684a68ed674acd" Nov 03 21:14:44.038813 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:44.038711 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd4421526c4ecde741d5b2716623486237c2c7688cb46b1c2f684a68ed674acd"} err="failed to get container status \"fd4421526c4ecde741d5b2716623486237c2c7688cb46b1c2f684a68ed674acd\": rpc error: code = NotFound desc = could not find container \"fd4421526c4ecde741d5b2716623486237c2c7688cb46b1c2f684a68ed674acd\": container with ID starting with fd4421526c4ecde741d5b2716623486237c2c7688cb46b1c2f684a68ed674acd not found: ID does not exist" Nov 03 21:14:44.038813 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:44.038731 4305 scope.go:122] "RemoveContainer" containerID="1b30d77ad6e760a8dec5065402b6f262ec0231531ac296a326ee1ad0abfa0e98" Nov 03 21:14:44.051221 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:44.051191 4305 scope.go:122] "RemoveContainer" containerID="1b30d77ad6e760a8dec5065402b6f262ec0231531ac296a326ee1ad0abfa0e98" Nov 03 21:14:44.051674 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:14:44.051651 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b30d77ad6e760a8dec5065402b6f262ec0231531ac296a326ee1ad0abfa0e98\": container with ID starting with 1b30d77ad6e760a8dec5065402b6f262ec0231531ac296a326ee1ad0abfa0e98 not found: ID does not exist" containerID="1b30d77ad6e760a8dec5065402b6f262ec0231531ac296a326ee1ad0abfa0e98" Nov 03 21:14:44.051739 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:44.051689 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b30d77ad6e760a8dec5065402b6f262ec0231531ac296a326ee1ad0abfa0e98"} err="failed to get container status \"1b30d77ad6e760a8dec5065402b6f262ec0231531ac296a326ee1ad0abfa0e98\": rpc error: code = NotFound desc = could not find container \"1b30d77ad6e760a8dec5065402b6f262ec0231531ac296a326ee1ad0abfa0e98\": container with ID starting with 1b30d77ad6e760a8dec5065402b6f262ec0231531ac296a326ee1ad0abfa0e98 not found: ID does not exist" Nov 03 21:14:44.166052 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:44.166025 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn" Nov 03 21:14:44.167629 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:44.167609 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-862138-65fb688c66-fx4kk"] Nov 03 21:14:44.171902 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:44.171877 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-862138-65fb688c66-fx4kk"] Nov 03 21:14:44.246017 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:44.245984 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fea2251-d380-4c79-a754-1a3ec7bdf829-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fea2251-d380-4c79-a754-1a3ec7bdf829-istio-envoy\") pod \"2fea2251-d380-4c79-a754-1a3ec7bdf829\" (UID: \"2fea2251-d380-4c79-a754-1a3ec7bdf829\") " Nov 03 21:14:44.246186 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:44.246068 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fea2251-d380-4c79-a754-1a3ec7bdf829-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fea2251-d380-4c79-a754-1a3ec7bdf829-istiod-ca-cert\") pod \"2fea2251-d380-4c79-a754-1a3ec7bdf829\" (UID: \"2fea2251-d380-4c79-a754-1a3ec7bdf829\") " Nov 03 21:14:44.246186 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:44.246153 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fea2251-d380-4c79-a754-1a3ec7bdf829-kube-api-access-9qjnt\" (UniqueName: \"kubernetes.io/projected/2fea2251-d380-4c79-a754-1a3ec7bdf829-kube-api-access-9qjnt\") pod \"2fea2251-d380-4c79-a754-1a3ec7bdf829\" (UID: \"2fea2251-d380-4c79-a754-1a3ec7bdf829\") " Nov 03 21:14:44.246186 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:44.246181 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fea2251-d380-4c79-a754-1a3ec7bdf829-istio-token\" (UniqueName: \"kubernetes.io/projected/2fea2251-d380-4c79-a754-1a3ec7bdf829-istio-token\") pod \"2fea2251-d380-4c79-a754-1a3ec7bdf829\" (UID: \"2fea2251-d380-4c79-a754-1a3ec7bdf829\") " Nov 03 21:14:44.246370 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:44.246236 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fea2251-d380-4c79-a754-1a3ec7bdf829-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fea2251-d380-4c79-a754-1a3ec7bdf829-workload-socket\") pod \"2fea2251-d380-4c79-a754-1a3ec7bdf829\" (UID: \"2fea2251-d380-4c79-a754-1a3ec7bdf829\") " Nov 03 21:14:44.246370 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:44.246290 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2fea2251-d380-4c79-a754-1a3ec7bdf829-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fea2251-d380-4c79-a754-1a3ec7bdf829-istio-podinfo\") pod \"2fea2251-d380-4c79-a754-1a3ec7bdf829\" (UID: \"2fea2251-d380-4c79-a754-1a3ec7bdf829\") " Nov 03 21:14:44.246370 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:44.246325 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fea2251-d380-4c79-a754-1a3ec7bdf829-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fea2251-d380-4c79-a754-1a3ec7bdf829-credential-socket\") pod \"2fea2251-d380-4c79-a754-1a3ec7bdf829\" (UID: \"2fea2251-d380-4c79-a754-1a3ec7bdf829\") " Nov 03 21:14:44.246370 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:44.246368 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fea2251-d380-4c79-a754-1a3ec7bdf829-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fea2251-d380-4c79-a754-1a3ec7bdf829-istio-data\") pod \"2fea2251-d380-4c79-a754-1a3ec7bdf829\" (UID: \"2fea2251-d380-4c79-a754-1a3ec7bdf829\") " Nov 03 21:14:44.246652 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:44.246417 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fea2251-d380-4c79-a754-1a3ec7bdf829-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fea2251-d380-4c79-a754-1a3ec7bdf829-workload-certs\") pod \"2fea2251-d380-4c79-a754-1a3ec7bdf829\" (UID: \"2fea2251-d380-4c79-a754-1a3ec7bdf829\") " Nov 03 21:14:44.246652 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:44.246601 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fea2251-d380-4c79-a754-1a3ec7bdf829-workload-socket" pod "2fea2251-d380-4c79-a754-1a3ec7bdf829" (UID: "2fea2251-d380-4c79-a754-1a3ec7bdf829"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:44.246780 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:44.246660 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fea2251-d380-4c79-a754-1a3ec7bdf829-istiod-ca-cert" pod "2fea2251-d380-4c79-a754-1a3ec7bdf829" (UID: "2fea2251-d380-4c79-a754-1a3ec7bdf829"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:14:44.246780 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:44.246681 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fea2251-d380-4c79-a754-1a3ec7bdf829-credential-socket" pod "2fea2251-d380-4c79-a754-1a3ec7bdf829" (UID: "2fea2251-d380-4c79-a754-1a3ec7bdf829"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:44.246889 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:44.246824 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fea2251-d380-4c79-a754-1a3ec7bdf829-workload-certs" pod "2fea2251-d380-4c79-a754-1a3ec7bdf829" (UID: "2fea2251-d380-4c79-a754-1a3ec7bdf829"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:44.246889 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:44.246854 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fea2251-d380-4c79-a754-1a3ec7bdf829-istio-data" pod "2fea2251-d380-4c79-a754-1a3ec7bdf829" (UID: "2fea2251-d380-4c79-a754-1a3ec7bdf829"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:44.247169 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:44.246937 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fea2251-d380-4c79-a754-1a3ec7bdf829-workload-certs\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:44.247169 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:44.246962 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fea2251-d380-4c79-a754-1a3ec7bdf829-istiod-ca-cert\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:44.247169 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:44.246976 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fea2251-d380-4c79-a754-1a3ec7bdf829-workload-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:44.247169 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:44.246989 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fea2251-d380-4c79-a754-1a3ec7bdf829-credential-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:44.248801 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:44.248772 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fea2251-d380-4c79-a754-1a3ec7bdf829-kube-api-access-9qjnt" pod "2fea2251-d380-4c79-a754-1a3ec7bdf829" (UID: "2fea2251-d380-4c79-a754-1a3ec7bdf829"). InnerVolumeSpecName "kube-api-access-9qjnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:14:44.248801 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:44.248782 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fea2251-d380-4c79-a754-1a3ec7bdf829-istio-token" pod "2fea2251-d380-4c79-a754-1a3ec7bdf829" (UID: "2fea2251-d380-4c79-a754-1a3ec7bdf829"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:14:44.249145 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:44.249126 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fea2251-d380-4c79-a754-1a3ec7bdf829-istio-envoy" pod "2fea2251-d380-4c79-a754-1a3ec7bdf829" (UID: "2fea2251-d380-4c79-a754-1a3ec7bdf829"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:44.249145 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:44.249132 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2fea2251-d380-4c79-a754-1a3ec7bdf829-istio-podinfo" pod "2fea2251-d380-4c79-a754-1a3ec7bdf829" (UID: "2fea2251-d380-4c79-a754-1a3ec7bdf829"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:14:44.282143 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:44.282110 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-862138-6c8d9c5567-7nvj6"] Nov 03 21:14:44.288365 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:44.288338 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-862138-6c8d9c5567-7nvj6"] Nov 03 21:14:44.348767 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:44.348730 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fea2251-d380-4c79-a754-1a3ec7bdf829-istio-data\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:44.348767 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:44.348768 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fea2251-d380-4c79-a754-1a3ec7bdf829-istio-envoy\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:44.349022 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:44.348787 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qjnt\" (UniqueName: \"kubernetes.io/projected/2fea2251-d380-4c79-a754-1a3ec7bdf829-kube-api-access-9qjnt\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:44.349022 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:44.348803 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fea2251-d380-4c79-a754-1a3ec7bdf829-istio-token\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:44.349022 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:44.348816 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fea2251-d380-4c79-a754-1a3ec7bdf829-istio-podinfo\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:44.894801 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:44.894769 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e3a291b-d5be-4d9c-8b8a-0b97dba864e2" path="/var/lib/kubelet/pods/3e3a291b-d5be-4d9c-8b8a-0b97dba864e2/volumes" Nov 03 21:14:44.895200 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:44.895130 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c3bdfd9-c058-49d6-9179-4a53b523216f" path="/var/lib/kubelet/pods/6c3bdfd9-c058-49d6-9179-4a53b523216f/volumes" Nov 03 21:14:44.958796 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:44.958760 4305 generic.go:356] "Generic (PLEG): container finished" podID="2fea2251-d380-4c79-a754-1a3ec7bdf829" containerID="caa40f2d41760bebe3ffe2434ec0064e45f551b4431a64c9fd2351ec5f854170" exitCode=0 Nov 03 21:14:44.958968 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:44.958809 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn" Nov 03 21:14:44.958968 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:44.958847 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn" event={"ID":"2fea2251-d380-4c79-a754-1a3ec7bdf829","Type":"ContainerDied","Data":"caa40f2d41760bebe3ffe2434ec0064e45f551b4431a64c9fd2351ec5f854170"} Nov 03 21:14:44.958968 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:44.958907 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn" event={"ID":"2fea2251-d380-4c79-a754-1a3ec7bdf829","Type":"ContainerDied","Data":"ba31d07d3ff03871282fc2e775febd2637614fa930d20f55d233be381d08782a"} Nov 03 21:14:44.958968 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:44.958929 4305 scope.go:122] "RemoveContainer" containerID="caa40f2d41760bebe3ffe2434ec0064e45f551b4431a64c9fd2351ec5f854170" Nov 03 21:14:44.969514 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:44.969492 4305 scope.go:122] "RemoveContainer" containerID="caa40f2d41760bebe3ffe2434ec0064e45f551b4431a64c9fd2351ec5f854170" Nov 03 21:14:44.969789 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:14:44.969770 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"caa40f2d41760bebe3ffe2434ec0064e45f551b4431a64c9fd2351ec5f854170\": container with ID starting with caa40f2d41760bebe3ffe2434ec0064e45f551b4431a64c9fd2351ec5f854170 not found: ID does not exist" containerID="caa40f2d41760bebe3ffe2434ec0064e45f551b4431a64c9fd2351ec5f854170" Nov 03 21:14:44.969837 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:44.969799 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caa40f2d41760bebe3ffe2434ec0064e45f551b4431a64c9fd2351ec5f854170"} err="failed to get container status \"caa40f2d41760bebe3ffe2434ec0064e45f551b4431a64c9fd2351ec5f854170\": rpc error: code = NotFound desc = could not find container \"caa40f2d41760bebe3ffe2434ec0064e45f551b4431a64c9fd2351ec5f854170\": container with ID starting with caa40f2d41760bebe3ffe2434ec0064e45f551b4431a64c9fd2351ec5f854170 not found: ID does not exist" Nov 03 21:14:45.124585 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:45.124549 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn"] Nov 03 21:14:45.173679 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:45.173601 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-862138-openshift-default-bc56c55fc-v4ggn"] Nov 03 21:14:46.185357 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:46.185332 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr" Nov 03 21:14:46.268467 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:46.268361 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbb09c2d-a432-4507-a7d2-9e401f0e96bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbb09c2d-a432-4507-a7d2-9e401f0e96bb-workload-certs\") pod \"fbb09c2d-a432-4507-a7d2-9e401f0e96bb\" (UID: \"fbb09c2d-a432-4507-a7d2-9e401f0e96bb\") " Nov 03 21:14:46.268467 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:46.268446 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbb09c2d-a432-4507-a7d2-9e401f0e96bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbb09c2d-a432-4507-a7d2-9e401f0e96bb-istiod-ca-cert\") pod \"fbb09c2d-a432-4507-a7d2-9e401f0e96bb\" (UID: \"fbb09c2d-a432-4507-a7d2-9e401f0e96bb\") " Nov 03 21:14:46.268705 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:46.268476 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbb09c2d-a432-4507-a7d2-9e401f0e96bb-kube-api-access-x7ngw\" (UniqueName: \"kubernetes.io/projected/fbb09c2d-a432-4507-a7d2-9e401f0e96bb-kube-api-access-x7ngw\") pod \"fbb09c2d-a432-4507-a7d2-9e401f0e96bb\" (UID: \"fbb09c2d-a432-4507-a7d2-9e401f0e96bb\") " Nov 03 21:14:46.268705 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:46.268514 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbb09c2d-a432-4507-a7d2-9e401f0e96bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb09c2d-a432-4507-a7d2-9e401f0e96bb-credential-socket\") pod \"fbb09c2d-a432-4507-a7d2-9e401f0e96bb\" (UID: \"fbb09c2d-a432-4507-a7d2-9e401f0e96bb\") " Nov 03 21:14:46.268705 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:46.268561 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbb09c2d-a432-4507-a7d2-9e401f0e96bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb09c2d-a432-4507-a7d2-9e401f0e96bb-workload-socket\") pod \"fbb09c2d-a432-4507-a7d2-9e401f0e96bb\" (UID: \"fbb09c2d-a432-4507-a7d2-9e401f0e96bb\") " Nov 03 21:14:46.268705 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:46.268582 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbb09c2d-a432-4507-a7d2-9e401f0e96bb-istio-token\" (UniqueName: \"kubernetes.io/projected/fbb09c2d-a432-4507-a7d2-9e401f0e96bb-istio-token\") pod \"fbb09c2d-a432-4507-a7d2-9e401f0e96bb\" (UID: \"fbb09c2d-a432-4507-a7d2-9e401f0e96bb\") " Nov 03 21:14:46.268705 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:46.268664 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbb09c2d-a432-4507-a7d2-9e401f0e96bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbb09c2d-a432-4507-a7d2-9e401f0e96bb-istio-envoy\") pod \"fbb09c2d-a432-4507-a7d2-9e401f0e96bb\" (UID: \"fbb09c2d-a432-4507-a7d2-9e401f0e96bb\") " Nov 03 21:14:46.268705 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:46.268670 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbb09c2d-a432-4507-a7d2-9e401f0e96bb-workload-certs" pod "fbb09c2d-a432-4507-a7d2-9e401f0e96bb" (UID: "fbb09c2d-a432-4507-a7d2-9e401f0e96bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:46.268705 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:46.268696 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbb09c2d-a432-4507-a7d2-9e401f0e96bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbb09c2d-a432-4507-a7d2-9e401f0e96bb-istio-data\") pod \"fbb09c2d-a432-4507-a7d2-9e401f0e96bb\" (UID: \"fbb09c2d-a432-4507-a7d2-9e401f0e96bb\") " Nov 03 21:14:46.269123 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:46.268728 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fbb09c2d-a432-4507-a7d2-9e401f0e96bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbb09c2d-a432-4507-a7d2-9e401f0e96bb-istio-podinfo\") pod \"fbb09c2d-a432-4507-a7d2-9e401f0e96bb\" (UID: \"fbb09c2d-a432-4507-a7d2-9e401f0e96bb\") " Nov 03 21:14:46.269123 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:46.268872 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbb09c2d-a432-4507-a7d2-9e401f0e96bb-workload-socket" pod "fbb09c2d-a432-4507-a7d2-9e401f0e96bb" (UID: "fbb09c2d-a432-4507-a7d2-9e401f0e96bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:46.269123 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:46.268942 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbb09c2d-a432-4507-a7d2-9e401f0e96bb-istio-data" pod "fbb09c2d-a432-4507-a7d2-9e401f0e96bb" (UID: "fbb09c2d-a432-4507-a7d2-9e401f0e96bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:46.269123 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:46.269060 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbb09c2d-a432-4507-a7d2-9e401f0e96bb-istio-data\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:46.269123 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:46.269080 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbb09c2d-a432-4507-a7d2-9e401f0e96bb-workload-certs\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:46.269123 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:46.269096 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb09c2d-a432-4507-a7d2-9e401f0e96bb-workload-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:46.269123 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:46.269105 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbb09c2d-a432-4507-a7d2-9e401f0e96bb-istiod-ca-cert" pod "fbb09c2d-a432-4507-a7d2-9e401f0e96bb" (UID: "fbb09c2d-a432-4507-a7d2-9e401f0e96bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:14:46.269455 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:46.269160 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbb09c2d-a432-4507-a7d2-9e401f0e96bb-credential-socket" pod "fbb09c2d-a432-4507-a7d2-9e401f0e96bb" (UID: "fbb09c2d-a432-4507-a7d2-9e401f0e96bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:46.271045 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:46.271018 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbb09c2d-a432-4507-a7d2-9e401f0e96bb-istio-envoy" pod "fbb09c2d-a432-4507-a7d2-9e401f0e96bb" (UID: "fbb09c2d-a432-4507-a7d2-9e401f0e96bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:46.271165 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:46.271021 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fbb09c2d-a432-4507-a7d2-9e401f0e96bb-istio-podinfo" pod "fbb09c2d-a432-4507-a7d2-9e401f0e96bb" (UID: "fbb09c2d-a432-4507-a7d2-9e401f0e96bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:14:46.271331 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:46.271311 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbb09c2d-a432-4507-a7d2-9e401f0e96bb-istio-token" pod "fbb09c2d-a432-4507-a7d2-9e401f0e96bb" (UID: "fbb09c2d-a432-4507-a7d2-9e401f0e96bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:14:46.271462 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:46.271333 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbb09c2d-a432-4507-a7d2-9e401f0e96bb-kube-api-access-x7ngw" pod "fbb09c2d-a432-4507-a7d2-9e401f0e96bb" (UID: "fbb09c2d-a432-4507-a7d2-9e401f0e96bb"). InnerVolumeSpecName "kube-api-access-x7ngw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:14:46.320600 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:14:46.320568 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:14:46.370428 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:46.370400 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbb09c2d-a432-4507-a7d2-9e401f0e96bb-istio-envoy\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:46.370640 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:46.370428 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbb09c2d-a432-4507-a7d2-9e401f0e96bb-istio-podinfo\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:46.370640 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:46.370550 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbb09c2d-a432-4507-a7d2-9e401f0e96bb-istiod-ca-cert\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:46.370640 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:46.370563 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7ngw\" (UniqueName: \"kubernetes.io/projected/fbb09c2d-a432-4507-a7d2-9e401f0e96bb-kube-api-access-x7ngw\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:46.370640 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:46.370576 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb09c2d-a432-4507-a7d2-9e401f0e96bb-credential-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:46.370640 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:46.370591 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbb09c2d-a432-4507-a7d2-9e401f0e96bb-istio-token\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:14:46.895258 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:46.895223 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fea2251-d380-4c79-a754-1a3ec7bdf829" path="/var/lib/kubelet/pods/2fea2251-d380-4c79-a754-1a3ec7bdf829/volumes" Nov 03 21:14:46.978247 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:46.978202 4305 generic.go:356] "Generic (PLEG): container finished" podID="fbb09c2d-a432-4507-a7d2-9e401f0e96bb" containerID="c3207900e035579b7be53a7527c6e070412de23241a06f4490a143d042b544e6" exitCode=0 Nov 03 21:14:46.978463 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:46.978311 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr" Nov 03 21:14:46.978463 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:46.978349 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr" event={"ID":"fbb09c2d-a432-4507-a7d2-9e401f0e96bb","Type":"ContainerDied","Data":"c3207900e035579b7be53a7527c6e070412de23241a06f4490a143d042b544e6"} Nov 03 21:14:46.978463 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:46.978412 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr" event={"ID":"fbb09c2d-a432-4507-a7d2-9e401f0e96bb","Type":"ContainerDied","Data":"c94c4e5ffe85e526f51e218f76e4629548e8149b0421204e528fc41bfc8ccbad"} Nov 03 21:14:46.978627 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:46.978472 4305 scope.go:122] "RemoveContainer" containerID="c3207900e035579b7be53a7527c6e070412de23241a06f4490a143d042b544e6" Nov 03 21:14:46.988756 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:46.988736 4305 scope.go:122] "RemoveContainer" containerID="c3207900e035579b7be53a7527c6e070412de23241a06f4490a143d042b544e6" Nov 03 21:14:46.989043 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:14:46.989021 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3207900e035579b7be53a7527c6e070412de23241a06f4490a143d042b544e6\": container with ID starting with c3207900e035579b7be53a7527c6e070412de23241a06f4490a143d042b544e6 not found: ID does not exist" containerID="c3207900e035579b7be53a7527c6e070412de23241a06f4490a143d042b544e6" Nov 03 21:14:46.989149 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:46.989050 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3207900e035579b7be53a7527c6e070412de23241a06f4490a143d042b544e6"} err="failed to get container status \"c3207900e035579b7be53a7527c6e070412de23241a06f4490a143d042b544e6\": rpc error: code = NotFound desc = could not find container \"c3207900e035579b7be53a7527c6e070412de23241a06f4490a143d042b544e6\": container with ID starting with c3207900e035579b7be53a7527c6e070412de23241a06f4490a143d042b544e6 not found: ID does not exist" Nov 03 21:14:47.143402 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:47.143371 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr"] Nov 03 21:14:47.192748 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:47.192676 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-12a423-openshift-default-747695c87f-b5srr"] Nov 03 21:14:48.895195 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:48.895165 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbb09c2d-a432-4507-a7d2-9e401f0e96bb" path="/var/lib/kubelet/pods/fbb09c2d-a432-4507-a7d2-9e401f0e96bb/volumes" Nov 03 21:14:49.048424 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:49.048388 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6349b785-88d6n"] Nov 03 21:14:49.098144 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:49.098089 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6349b785-88d6n"] Nov 03 21:14:49.100426 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:49.100398 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5769a2cf-rmwbr"] Nov 03 21:14:49.103319 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:49.103299 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5769a2cf-rmwbr"] Nov 03 21:14:50.894387 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:50.894340 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c31b220a-debb-4e97-9dd9-03c304faf447" path="/var/lib/kubelet/pods/c31b220a-debb-4e97-9dd9-03c304faf447/volumes" Nov 03 21:14:50.894981 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:14:50.894960 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6f56404-e1ad-4521-b8ba-ab22614197a0" path="/var/lib/kubelet/pods/c6f56404-e1ad-4521-b8ba-ab22614197a0/volumes" Nov 03 21:15:06.492311 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:06.492270 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:15:11.269511 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:11.269471 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-trmz9"] Nov 03 21:15:11.270135 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:11.270117 4305 cpu_manager.go:414] "containerMap: removing container" podUID="fbb09c2d-a432-4507-a7d2-9e401f0e96bb" containerName="istio-proxy" Nov 03 21:15:11.270181 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:11.270139 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbb09c2d-a432-4507-a7d2-9e401f0e96bb" containerName="istio-proxy" Nov 03 21:15:11.270181 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:11.270162 4305 cpu_manager.go:414] "containerMap: removing container" podUID="6c3bdfd9-c058-49d6-9179-4a53b523216f" containerName="payload-processing" Nov 03 21:15:11.270181 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:11.270171 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c3bdfd9-c058-49d6-9179-4a53b523216f" containerName="payload-processing" Nov 03 21:15:11.270285 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:11.270197 4305 cpu_manager.go:414] "containerMap: removing container" podUID="3e6b8d49-0db7-4bbc-9b3e-1105d2bdbc78" containerName="payload-processing" Nov 03 21:15:11.270285 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:11.270206 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e6b8d49-0db7-4bbc-9b3e-1105d2bdbc78" containerName="payload-processing" Nov 03 21:15:11.270285 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:11.270217 4305 cpu_manager.go:414] "containerMap: removing container" podUID="3e3a291b-d5be-4d9c-8b8a-0b97dba864e2" containerName="payload-pre-processing" Nov 03 21:15:11.270285 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:11.270226 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e3a291b-d5be-4d9c-8b8a-0b97dba864e2" containerName="payload-pre-processing" Nov 03 21:15:11.270285 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:11.270240 4305 cpu_manager.go:414] "containerMap: removing container" podUID="a20ce38b-39a2-4e14-92f9-ad37dd2ad124" containerName="revoke-keys" Nov 03 21:15:11.270285 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:11.270249 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a20ce38b-39a2-4e14-92f9-ad37dd2ad124" containerName="revoke-keys" Nov 03 21:15:11.270285 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:11.270270 4305 cpu_manager.go:414] "containerMap: removing container" podUID="2fea2251-d380-4c79-a754-1a3ec7bdf829" containerName="istio-proxy" Nov 03 21:15:11.270285 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:11.270278 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fea2251-d380-4c79-a754-1a3ec7bdf829" containerName="istio-proxy" Nov 03 21:15:11.270574 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:11.270289 4305 cpu_manager.go:414] "containerMap: removing container" podUID="a2e9589d-058b-47fb-a28d-9b92a855c9ca" containerName="revoke-keys" Nov 03 21:15:11.270574 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:11.270298 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2e9589d-058b-47fb-a28d-9b92a855c9ca" containerName="revoke-keys" Nov 03 21:15:11.270574 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:11.270311 4305 cpu_manager.go:414] "containerMap: removing container" podUID="7b0135ee-b547-46e8-9953-6a46cb61f2c4" containerName="payload-pre-processing" Nov 03 21:15:11.270574 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:11.270319 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b0135ee-b547-46e8-9953-6a46cb61f2c4" containerName="payload-pre-processing" Nov 03 21:15:11.270574 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:11.270336 4305 cpu_manager.go:414] "containerMap: removing container" podUID="b24ff37e-1709-4f11-b0e8-9302b9f3d9ad" containerName="istio-proxy" Nov 03 21:15:11.270574 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:11.270344 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b24ff37e-1709-4f11-b0e8-9302b9f3d9ad" containerName="istio-proxy" Nov 03 21:15:11.270574 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:11.270464 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbb09c2d-a432-4507-a7d2-9e401f0e96bb" containerName="istio-proxy" Nov 03 21:15:11.270574 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:11.270479 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c3bdfd9-c058-49d6-9179-4a53b523216f" containerName="payload-processing" Nov 03 21:15:11.270574 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:11.270493 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e6b8d49-0db7-4bbc-9b3e-1105d2bdbc78" containerName="payload-processing" Nov 03 21:15:11.270574 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:11.270501 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e3a291b-d5be-4d9c-8b8a-0b97dba864e2" containerName="payload-pre-processing" Nov 03 21:15:11.270574 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:11.270510 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="a20ce38b-39a2-4e14-92f9-ad37dd2ad124" containerName="revoke-keys" Nov 03 21:15:11.270574 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:11.270520 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fea2251-d380-4c79-a754-1a3ec7bdf829" containerName="istio-proxy" Nov 03 21:15:11.270574 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:11.270529 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2e9589d-058b-47fb-a28d-9b92a855c9ca" containerName="revoke-keys" Nov 03 21:15:11.270574 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:11.270536 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b0135ee-b547-46e8-9953-6a46cb61f2c4" containerName="payload-pre-processing" Nov 03 21:15:11.270574 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:11.270546 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="b24ff37e-1709-4f11-b0e8-9302b9f3d9ad" containerName="istio-proxy" Nov 03 21:15:11.275692 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:11.275670 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-trmz9" Nov 03 21:15:11.329229 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:11.329190 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-trmz9"] Nov 03 21:15:11.374972 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:11.374923 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:15:11.374972 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:11.374954 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:15:11.374972 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:11.374954 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77vtd\"" Nov 03 21:15:11.377578 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:11.377558 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:15:11.423115 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:11.423081 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44bb69ae-9081-4986-a048-5fe5c3c89ff4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-trmz9\" (UID: \"44bb69ae-9081-4986-a048-5fe5c3c89ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-trmz9" Nov 03 21:15:11.423311 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:11.423151 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44bb69ae-9081-4986-a048-5fe5c3c89ff4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-trmz9\" (UID: \"44bb69ae-9081-4986-a048-5fe5c3c89ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-trmz9" Nov 03 21:15:11.423311 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:11.423273 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8w7v\" (UniqueName: \"kubernetes.io/projected/44bb69ae-9081-4986-a048-5fe5c3c89ff4-kube-api-access-v8w7v\") pod \"maas-api-revoke-sub-85237e45-trmz9\" (UID: \"44bb69ae-9081-4986-a048-5fe5c3c89ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-trmz9" Nov 03 21:15:11.523992 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:11.523894 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44bb69ae-9081-4986-a048-5fe5c3c89ff4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-trmz9\" (UID: \"44bb69ae-9081-4986-a048-5fe5c3c89ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-trmz9" Nov 03 21:15:11.523992 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:11.523961 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8w7v\" (UniqueName: \"kubernetes.io/projected/44bb69ae-9081-4986-a048-5fe5c3c89ff4-kube-api-access-v8w7v\") pod \"maas-api-revoke-sub-85237e45-trmz9\" (UID: \"44bb69ae-9081-4986-a048-5fe5c3c89ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-trmz9" Nov 03 21:15:11.524201 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:11.524039 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44bb69ae-9081-4986-a048-5fe5c3c89ff4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-trmz9\" (UID: \"44bb69ae-9081-4986-a048-5fe5c3c89ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-trmz9" Nov 03 21:15:11.524591 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:11.524559 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44bb69ae-9081-4986-a048-5fe5c3c89ff4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-trmz9\" (UID: \"44bb69ae-9081-4986-a048-5fe5c3c89ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-trmz9" Nov 03 21:15:11.526626 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:11.526603 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44bb69ae-9081-4986-a048-5fe5c3c89ff4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-trmz9\" (UID: \"44bb69ae-9081-4986-a048-5fe5c3c89ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-trmz9" Nov 03 21:15:11.585831 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:11.585800 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8w7v\" (UniqueName: \"kubernetes.io/projected/44bb69ae-9081-4986-a048-5fe5c3c89ff4-kube-api-access-v8w7v\") pod \"maas-api-revoke-sub-85237e45-trmz9\" (UID: \"44bb69ae-9081-4986-a048-5fe5c3c89ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-trmz9" Nov 03 21:15:11.599379 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:11.599351 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-trmz9" Nov 03 21:15:11.976428 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:11.976393 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-trmz9"] Nov 03 21:15:11.979887 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:15:11.979856 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44bb69ae_9081_4986_a048_5fe5c3c89ff4.slice/crio-1d7be5306f6053465a0ac607d8a842983cdde3cac0e803b065c58dd331dde7e9 WatchSource:0}: Error finding container 1d7be5306f6053465a0ac607d8a842983cdde3cac0e803b065c58dd331dde7e9: Status 404 returned error can't find the container with id 1d7be5306f6053465a0ac607d8a842983cdde3cac0e803b065c58dd331dde7e9 Nov 03 21:15:12.141001 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:12.140971 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-trmz9" event={"ID":"44bb69ae-9081-4986-a048-5fe5c3c89ff4","Type":"ContainerStarted","Data":"1d7be5306f6053465a0ac607d8a842983cdde3cac0e803b065c58dd331dde7e9"} Nov 03 21:15:13.148632 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:13.148592 4305 generic.go:356] "Generic (PLEG): container finished" podID="44bb69ae-9081-4986-a048-5fe5c3c89ff4" containerID="d77c119af204499e3ffd6914d0304c5e7b0b65ef421ece23d6d32a2a05a9b362" exitCode=0 Nov 03 21:15:13.149085 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:13.148681 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-trmz9" event={"ID":"44bb69ae-9081-4986-a048-5fe5c3c89ff4","Type":"ContainerDied","Data":"d77c119af204499e3ffd6914d0304c5e7b0b65ef421ece23d6d32a2a05a9b362"} Nov 03 21:15:14.295387 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:14.295361 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-trmz9" Nov 03 21:15:14.349647 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:14.349615 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44bb69ae-9081-4986-a048-5fe5c3c89ff4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44bb69ae-9081-4986-a048-5fe5c3c89ff4-maas-api-xks-ca\") pod \"44bb69ae-9081-4986-a048-5fe5c3c89ff4\" (UID: \"44bb69ae-9081-4986-a048-5fe5c3c89ff4\") " Nov 03 21:15:14.349647 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:14.349648 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44bb69ae-9081-4986-a048-5fe5c3c89ff4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44bb69ae-9081-4986-a048-5fe5c3c89ff4-maas-api-openshift-ca\") pod \"44bb69ae-9081-4986-a048-5fe5c3c89ff4\" (UID: \"44bb69ae-9081-4986-a048-5fe5c3c89ff4\") " Nov 03 21:15:14.349919 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:14.349702 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44bb69ae-9081-4986-a048-5fe5c3c89ff4-kube-api-access-v8w7v\" (UniqueName: \"kubernetes.io/projected/44bb69ae-9081-4986-a048-5fe5c3c89ff4-kube-api-access-v8w7v\") pod \"44bb69ae-9081-4986-a048-5fe5c3c89ff4\" (UID: \"44bb69ae-9081-4986-a048-5fe5c3c89ff4\") " Nov 03 21:15:14.350092 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:14.350068 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44bb69ae-9081-4986-a048-5fe5c3c89ff4-maas-api-openshift-ca" pod "44bb69ae-9081-4986-a048-5fe5c3c89ff4" (UID: "44bb69ae-9081-4986-a048-5fe5c3c89ff4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:15:14.351962 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:14.351940 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44bb69ae-9081-4986-a048-5fe5c3c89ff4-kube-api-access-v8w7v" pod "44bb69ae-9081-4986-a048-5fe5c3c89ff4" (UID: "44bb69ae-9081-4986-a048-5fe5c3c89ff4"). InnerVolumeSpecName "kube-api-access-v8w7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:15:14.352128 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:14.352109 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44bb69ae-9081-4986-a048-5fe5c3c89ff4-maas-api-xks-ca" pod "44bb69ae-9081-4986-a048-5fe5c3c89ff4" (UID: "44bb69ae-9081-4986-a048-5fe5c3c89ff4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:15:14.450537 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:14.450459 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44bb69ae-9081-4986-a048-5fe5c3c89ff4-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:15:14.450537 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:14.450489 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44bb69ae-9081-4986-a048-5fe5c3c89ff4-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:15:14.450537 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:14.450499 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8w7v\" (UniqueName: \"kubernetes.io/projected/44bb69ae-9081-4986-a048-5fe5c3c89ff4-kube-api-access-v8w7v\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:15:15.166582 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:15.166495 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-trmz9" Nov 03 21:15:15.166582 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:15.166525 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-trmz9" event={"ID":"44bb69ae-9081-4986-a048-5fe5c3c89ff4","Type":"ContainerDied","Data":"1d7be5306f6053465a0ac607d8a842983cdde3cac0e803b065c58dd331dde7e9"} Nov 03 21:15:15.166785 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:15.166598 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d7be5306f6053465a0ac607d8a842983cdde3cac0e803b065c58dd331dde7e9" Nov 03 21:15:16.321829 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:15:16.321788 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:15:27.786356 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.786318 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz"] Nov 03 21:15:27.786863 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.786739 4305 cpu_manager.go:414] "containerMap: removing container" podUID="44bb69ae-9081-4986-a048-5fe5c3c89ff4" containerName="revoke-keys" Nov 03 21:15:27.786863 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.786751 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44bb69ae-9081-4986-a048-5fe5c3c89ff4" containerName="revoke-keys" Nov 03 21:15:27.786863 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.786828 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="44bb69ae-9081-4986-a048-5fe5c3c89ff4" containerName="revoke-keys" Nov 03 21:15:27.789884 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.789866 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz" Nov 03 21:15:27.836315 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.836280 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n"] Nov 03 21:15:27.839840 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.839816 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7"] Nov 03 21:15:27.840052 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.839954 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n" Nov 03 21:15:27.843286 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.843264 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7" Nov 03 21:15:27.882981 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.882945 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-mmsql\"" Nov 03 21:15:27.886415 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.886383 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e88a59e-f72d-4345-8fc1-5c7113dc4638-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n\" (UID: \"5e88a59e-f72d-4345-8fc1-5c7113dc4638\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n" Nov 03 21:15:27.886571 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.886450 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/308949ce-8a8c-4cd7-ac5a-a778aeaa4caf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7\" (UID: \"308949ce-8a8c-4cd7-ac5a-a778aeaa4caf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7" Nov 03 21:15:27.886630 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.886606 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e88a59e-f72d-4345-8fc1-5c7113dc4638-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n\" (UID: \"5e88a59e-f72d-4345-8fc1-5c7113dc4638\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n" Nov 03 21:15:27.886680 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.886648 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/308949ce-8a8c-4cd7-ac5a-a778aeaa4caf-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7\" (UID: \"308949ce-8a8c-4cd7-ac5a-a778aeaa4caf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7" Nov 03 21:15:27.886730 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.886675 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f872846-a625-48d9-af4e-5854d31069ab-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz\" (UID: \"9f872846-a625-48d9-af4e-5854d31069ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz" Nov 03 21:15:27.886794 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.886768 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f872846-a625-48d9-af4e-5854d31069ab-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz\" (UID: \"9f872846-a625-48d9-af4e-5854d31069ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz" Nov 03 21:15:27.886858 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.886811 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e88a59e-f72d-4345-8fc1-5c7113dc4638-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n\" (UID: \"5e88a59e-f72d-4345-8fc1-5c7113dc4638\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n" Nov 03 21:15:27.886858 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.886837 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e88a59e-f72d-4345-8fc1-5c7113dc4638-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n\" (UID: \"5e88a59e-f72d-4345-8fc1-5c7113dc4638\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n" Nov 03 21:15:27.886961 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.886937 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mhbx\" (UniqueName: \"kubernetes.io/projected/5e88a59e-f72d-4345-8fc1-5c7113dc4638-kube-api-access-4mhbx\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n\" (UID: \"5e88a59e-f72d-4345-8fc1-5c7113dc4638\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n" Nov 03 21:15:27.887002 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.886978 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e88a59e-f72d-4345-8fc1-5c7113dc4638-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n\" (UID: \"5e88a59e-f72d-4345-8fc1-5c7113dc4638\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n" Nov 03 21:15:27.887036 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.887003 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/308949ce-8a8c-4cd7-ac5a-a778aeaa4caf-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7\" (UID: \"308949ce-8a8c-4cd7-ac5a-a778aeaa4caf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7" Nov 03 21:15:27.887036 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.887024 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e88a59e-f72d-4345-8fc1-5c7113dc4638-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n\" (UID: \"5e88a59e-f72d-4345-8fc1-5c7113dc4638\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n" Nov 03 21:15:27.887146 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.887077 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f872846-a625-48d9-af4e-5854d31069ab-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz\" (UID: \"9f872846-a625-48d9-af4e-5854d31069ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz" Nov 03 21:15:27.887146 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.887109 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e88a59e-f72d-4345-8fc1-5c7113dc4638-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n\" (UID: \"5e88a59e-f72d-4345-8fc1-5c7113dc4638\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n" Nov 03 21:15:27.887236 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.887144 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f872846-a625-48d9-af4e-5854d31069ab-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz\" (UID: \"9f872846-a625-48d9-af4e-5854d31069ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz" Nov 03 21:15:27.887236 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.887201 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f872846-a625-48d9-af4e-5854d31069ab-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz\" (UID: \"9f872846-a625-48d9-af4e-5854d31069ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz" Nov 03 21:15:27.887315 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.887267 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/308949ce-8a8c-4cd7-ac5a-a778aeaa4caf-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7\" (UID: \"308949ce-8a8c-4cd7-ac5a-a778aeaa4caf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7" Nov 03 21:15:27.887315 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.887296 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f872846-a625-48d9-af4e-5854d31069ab-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz\" (UID: \"9f872846-a625-48d9-af4e-5854d31069ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz" Nov 03 21:15:27.887403 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.887320 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f872846-a625-48d9-af4e-5854d31069ab-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz\" (UID: \"9f872846-a625-48d9-af4e-5854d31069ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz" Nov 03 21:15:27.887489 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.887415 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkwl8\" (UniqueName: \"kubernetes.io/projected/9f872846-a625-48d9-af4e-5854d31069ab-kube-api-access-tkwl8\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz\" (UID: \"9f872846-a625-48d9-af4e-5854d31069ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz" Nov 03 21:15:27.887558 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.887493 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/308949ce-8a8c-4cd7-ac5a-a778aeaa4caf-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7\" (UID: \"308949ce-8a8c-4cd7-ac5a-a778aeaa4caf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7" Nov 03 21:15:27.887558 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.887516 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/308949ce-8a8c-4cd7-ac5a-a778aeaa4caf-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7\" (UID: \"308949ce-8a8c-4cd7-ac5a-a778aeaa4caf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7" Nov 03 21:15:27.887558 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.887540 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e88a59e-f72d-4345-8fc1-5c7113dc4638-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n\" (UID: \"5e88a59e-f72d-4345-8fc1-5c7113dc4638\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n" Nov 03 21:15:27.887696 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.887656 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/308949ce-8a8c-4cd7-ac5a-a778aeaa4caf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7\" (UID: \"308949ce-8a8c-4cd7-ac5a-a778aeaa4caf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7" Nov 03 21:15:27.887744 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.887702 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr2bw\" (UniqueName: \"kubernetes.io/projected/308949ce-8a8c-4cd7-ac5a-a778aeaa4caf-kube-api-access-mr2bw\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7\" (UID: \"308949ce-8a8c-4cd7-ac5a-a778aeaa4caf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7" Nov 03 21:15:27.887807 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.887740 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f872846-a625-48d9-af4e-5854d31069ab-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz\" (UID: \"9f872846-a625-48d9-af4e-5854d31069ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz" Nov 03 21:15:27.887863 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.887811 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/308949ce-8a8c-4cd7-ac5a-a778aeaa4caf-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7\" (UID: \"308949ce-8a8c-4cd7-ac5a-a778aeaa4caf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7" Nov 03 21:15:27.894682 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.894659 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz"] Nov 03 21:15:27.941289 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.941243 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7"] Nov 03 21:15:27.945133 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.945097 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n"] Nov 03 21:15:27.988807 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.988772 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkwl8\" (UniqueName: \"kubernetes.io/projected/9f872846-a625-48d9-af4e-5854d31069ab-kube-api-access-tkwl8\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz\" (UID: \"9f872846-a625-48d9-af4e-5854d31069ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz" Nov 03 21:15:27.988807 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.988811 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/308949ce-8a8c-4cd7-ac5a-a778aeaa4caf-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7\" (UID: \"308949ce-8a8c-4cd7-ac5a-a778aeaa4caf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7" Nov 03 21:15:27.989037 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.988829 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/308949ce-8a8c-4cd7-ac5a-a778aeaa4caf-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7\" (UID: \"308949ce-8a8c-4cd7-ac5a-a778aeaa4caf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7" Nov 03 21:15:27.989037 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.988882 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e88a59e-f72d-4345-8fc1-5c7113dc4638-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n\" (UID: \"5e88a59e-f72d-4345-8fc1-5c7113dc4638\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n" Nov 03 21:15:27.989037 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.988920 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/308949ce-8a8c-4cd7-ac5a-a778aeaa4caf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7\" (UID: \"308949ce-8a8c-4cd7-ac5a-a778aeaa4caf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7" Nov 03 21:15:27.989037 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.988948 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr2bw\" (UniqueName: \"kubernetes.io/projected/308949ce-8a8c-4cd7-ac5a-a778aeaa4caf-kube-api-access-mr2bw\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7\" (UID: \"308949ce-8a8c-4cd7-ac5a-a778aeaa4caf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7" Nov 03 21:15:27.989037 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.988964 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f872846-a625-48d9-af4e-5854d31069ab-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz\" (UID: \"9f872846-a625-48d9-af4e-5854d31069ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz" Nov 03 21:15:27.989037 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.988984 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/308949ce-8a8c-4cd7-ac5a-a778aeaa4caf-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7\" (UID: \"308949ce-8a8c-4cd7-ac5a-a778aeaa4caf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7" Nov 03 21:15:27.989037 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.989017 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e88a59e-f72d-4345-8fc1-5c7113dc4638-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n\" (UID: \"5e88a59e-f72d-4345-8fc1-5c7113dc4638\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n" Nov 03 21:15:27.989481 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.989055 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/308949ce-8a8c-4cd7-ac5a-a778aeaa4caf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7\" (UID: \"308949ce-8a8c-4cd7-ac5a-a778aeaa4caf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7" Nov 03 21:15:27.989481 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.989117 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e88a59e-f72d-4345-8fc1-5c7113dc4638-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n\" (UID: \"5e88a59e-f72d-4345-8fc1-5c7113dc4638\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n" Nov 03 21:15:27.989481 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.989144 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/308949ce-8a8c-4cd7-ac5a-a778aeaa4caf-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7\" (UID: \"308949ce-8a8c-4cd7-ac5a-a778aeaa4caf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7" Nov 03 21:15:27.989481 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.989167 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f872846-a625-48d9-af4e-5854d31069ab-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz\" (UID: \"9f872846-a625-48d9-af4e-5854d31069ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz" Nov 03 21:15:27.989481 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.989200 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f872846-a625-48d9-af4e-5854d31069ab-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz\" (UID: \"9f872846-a625-48d9-af4e-5854d31069ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz" Nov 03 21:15:27.989481 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.989238 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e88a59e-f72d-4345-8fc1-5c7113dc4638-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n\" (UID: \"5e88a59e-f72d-4345-8fc1-5c7113dc4638\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n" Nov 03 21:15:27.989481 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.989269 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e88a59e-f72d-4345-8fc1-5c7113dc4638-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n\" (UID: \"5e88a59e-f72d-4345-8fc1-5c7113dc4638\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n" Nov 03 21:15:27.989481 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.989321 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mhbx\" (UniqueName: \"kubernetes.io/projected/5e88a59e-f72d-4345-8fc1-5c7113dc4638-kube-api-access-4mhbx\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n\" (UID: \"5e88a59e-f72d-4345-8fc1-5c7113dc4638\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n" Nov 03 21:15:27.989481 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.989350 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e88a59e-f72d-4345-8fc1-5c7113dc4638-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n\" (UID: \"5e88a59e-f72d-4345-8fc1-5c7113dc4638\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n" Nov 03 21:15:27.989481 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.989413 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/308949ce-8a8c-4cd7-ac5a-a778aeaa4caf-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7\" (UID: \"308949ce-8a8c-4cd7-ac5a-a778aeaa4caf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7" Nov 03 21:15:27.989481 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.989486 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e88a59e-f72d-4345-8fc1-5c7113dc4638-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n\" (UID: \"5e88a59e-f72d-4345-8fc1-5c7113dc4638\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n" Nov 03 21:15:27.990112 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.989537 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f872846-a625-48d9-af4e-5854d31069ab-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz\" (UID: \"9f872846-a625-48d9-af4e-5854d31069ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz" Nov 03 21:15:27.990112 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.989544 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e88a59e-f72d-4345-8fc1-5c7113dc4638-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n\" (UID: \"5e88a59e-f72d-4345-8fc1-5c7113dc4638\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n" Nov 03 21:15:27.990112 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.989564 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e88a59e-f72d-4345-8fc1-5c7113dc4638-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n\" (UID: \"5e88a59e-f72d-4345-8fc1-5c7113dc4638\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n" Nov 03 21:15:27.990112 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.989606 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f872846-a625-48d9-af4e-5854d31069ab-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz\" (UID: \"9f872846-a625-48d9-af4e-5854d31069ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz" Nov 03 21:15:27.990112 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.989615 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/308949ce-8a8c-4cd7-ac5a-a778aeaa4caf-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7\" (UID: \"308949ce-8a8c-4cd7-ac5a-a778aeaa4caf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7" Nov 03 21:15:27.990112 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.989795 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e88a59e-f72d-4345-8fc1-5c7113dc4638-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n\" (UID: \"5e88a59e-f72d-4345-8fc1-5c7113dc4638\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n" Nov 03 21:15:27.990112 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.989894 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e88a59e-f72d-4345-8fc1-5c7113dc4638-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n\" (UID: \"5e88a59e-f72d-4345-8fc1-5c7113dc4638\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n" Nov 03 21:15:27.990112 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.989926 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/308949ce-8a8c-4cd7-ac5a-a778aeaa4caf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7\" (UID: \"308949ce-8a8c-4cd7-ac5a-a778aeaa4caf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7" Nov 03 21:15:27.990112 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.989934 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/308949ce-8a8c-4cd7-ac5a-a778aeaa4caf-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7\" (UID: \"308949ce-8a8c-4cd7-ac5a-a778aeaa4caf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7" Nov 03 21:15:27.990112 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.990073 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f872846-a625-48d9-af4e-5854d31069ab-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz\" (UID: \"9f872846-a625-48d9-af4e-5854d31069ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz" Nov 03 21:15:27.990721 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.990238 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f872846-a625-48d9-af4e-5854d31069ab-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz\" (UID: \"9f872846-a625-48d9-af4e-5854d31069ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz" Nov 03 21:15:27.990721 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.990409 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f872846-a625-48d9-af4e-5854d31069ab-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz\" (UID: \"9f872846-a625-48d9-af4e-5854d31069ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz" Nov 03 21:15:27.990721 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.990490 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/308949ce-8a8c-4cd7-ac5a-a778aeaa4caf-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7\" (UID: \"308949ce-8a8c-4cd7-ac5a-a778aeaa4caf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7" Nov 03 21:15:27.990721 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.990522 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f872846-a625-48d9-af4e-5854d31069ab-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz\" (UID: \"9f872846-a625-48d9-af4e-5854d31069ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz" Nov 03 21:15:27.990721 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.990551 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f872846-a625-48d9-af4e-5854d31069ab-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz\" (UID: \"9f872846-a625-48d9-af4e-5854d31069ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz" Nov 03 21:15:27.990721 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.990598 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/308949ce-8a8c-4cd7-ac5a-a778aeaa4caf-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7\" (UID: \"308949ce-8a8c-4cd7-ac5a-a778aeaa4caf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7" Nov 03 21:15:27.990721 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.990711 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e88a59e-f72d-4345-8fc1-5c7113dc4638-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n\" (UID: \"5e88a59e-f72d-4345-8fc1-5c7113dc4638\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n" Nov 03 21:15:27.991114 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.990884 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e88a59e-f72d-4345-8fc1-5c7113dc4638-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n\" (UID: \"5e88a59e-f72d-4345-8fc1-5c7113dc4638\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n" Nov 03 21:15:27.991230 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.991210 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/308949ce-8a8c-4cd7-ac5a-a778aeaa4caf-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7\" (UID: \"308949ce-8a8c-4cd7-ac5a-a778aeaa4caf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7" Nov 03 21:15:27.991779 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.991725 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f872846-a625-48d9-af4e-5854d31069ab-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz\" (UID: \"9f872846-a625-48d9-af4e-5854d31069ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz" Nov 03 21:15:27.992305 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.992136 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f872846-a625-48d9-af4e-5854d31069ab-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz\" (UID: \"9f872846-a625-48d9-af4e-5854d31069ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz" Nov 03 21:15:27.992305 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.992252 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f872846-a625-48d9-af4e-5854d31069ab-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz\" (UID: \"9f872846-a625-48d9-af4e-5854d31069ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz" Nov 03 21:15:27.992914 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.992889 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f872846-a625-48d9-af4e-5854d31069ab-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz\" (UID: \"9f872846-a625-48d9-af4e-5854d31069ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz" Nov 03 21:15:27.993204 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.993175 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/308949ce-8a8c-4cd7-ac5a-a778aeaa4caf-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7\" (UID: \"308949ce-8a8c-4cd7-ac5a-a778aeaa4caf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7" Nov 03 21:15:27.993297 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.993254 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e88a59e-f72d-4345-8fc1-5c7113dc4638-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n\" (UID: \"5e88a59e-f72d-4345-8fc1-5c7113dc4638\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n" Nov 03 21:15:27.993517 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.993496 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e88a59e-f72d-4345-8fc1-5c7113dc4638-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n\" (UID: \"5e88a59e-f72d-4345-8fc1-5c7113dc4638\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n" Nov 03 21:15:27.993637 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.993618 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/308949ce-8a8c-4cd7-ac5a-a778aeaa4caf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7\" (UID: \"308949ce-8a8c-4cd7-ac5a-a778aeaa4caf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7" Nov 03 21:15:27.993685 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:27.993616 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f872846-a625-48d9-af4e-5854d31069ab-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz\" (UID: \"9f872846-a625-48d9-af4e-5854d31069ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz" Nov 03 21:15:28.053514 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:28.053366 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr2bw\" (UniqueName: \"kubernetes.io/projected/308949ce-8a8c-4cd7-ac5a-a778aeaa4caf-kube-api-access-mr2bw\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7\" (UID: \"308949ce-8a8c-4cd7-ac5a-a778aeaa4caf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7" Nov 03 21:15:28.053679 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:28.053610 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e88a59e-f72d-4345-8fc1-5c7113dc4638-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n\" (UID: \"5e88a59e-f72d-4345-8fc1-5c7113dc4638\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n" Nov 03 21:15:28.053876 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:28.053848 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkwl8\" (UniqueName: \"kubernetes.io/projected/9f872846-a625-48d9-af4e-5854d31069ab-kube-api-access-tkwl8\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz\" (UID: \"9f872846-a625-48d9-af4e-5854d31069ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz" Nov 03 21:15:28.054356 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:28.054332 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f872846-a625-48d9-af4e-5854d31069ab-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz\" (UID: \"9f872846-a625-48d9-af4e-5854d31069ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz" Nov 03 21:15:28.054356 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:28.054348 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/308949ce-8a8c-4cd7-ac5a-a778aeaa4caf-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7\" (UID: \"308949ce-8a8c-4cd7-ac5a-a778aeaa4caf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7" Nov 03 21:15:28.055087 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:28.055038 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mhbx\" (UniqueName: \"kubernetes.io/projected/5e88a59e-f72d-4345-8fc1-5c7113dc4638-kube-api-access-4mhbx\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n\" (UID: \"5e88a59e-f72d-4345-8fc1-5c7113dc4638\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n" Nov 03 21:15:28.055743 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:28.055668 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b11be461-gsj8r"] Nov 03 21:15:28.058146 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:28.058121 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-043ccaa3-j4fhl"] Nov 03 21:15:28.060104 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:28.060081 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b11be461-gsj8r"] Nov 03 21:15:28.062229 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:28.062207 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-043ccaa3-j4fhl"] Nov 03 21:15:28.103231 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:28.103197 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz" Nov 03 21:15:28.155477 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:28.155203 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n" Nov 03 21:15:28.163877 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:28.162241 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7" Nov 03 21:15:28.483698 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:28.483674 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz"] Nov 03 21:15:28.485091 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:15:28.485062 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f872846_a625_48d9_af4e_5854d31069ab.slice/crio-07314be4527e85add65454e85cc91e54886cdd35bb8d7f28cfe9ec6b37067d03 WatchSource:0}: Error finding container 07314be4527e85add65454e85cc91e54886cdd35bb8d7f28cfe9ec6b37067d03: Status 404 returned error can't find the container with id 07314be4527e85add65454e85cc91e54886cdd35bb8d7f28cfe9ec6b37067d03 Nov 03 21:15:28.487219 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:28.487186 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:15:28.487278 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:28.487256 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:15:28.487317 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:28.487285 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:15:28.602355 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:28.602071 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n"] Nov 03 21:15:28.605528 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:28.605392 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7"] Nov 03 21:15:28.605604 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:15:28.605525 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod308949ce_8a8c_4cd7_ac5a_a778aeaa4caf.slice/crio-29e185e0d944ef3124a2b576ef3b4cb665bfa43c37f126a33f86dcdf34328c8c WatchSource:0}: Error finding container 29e185e0d944ef3124a2b576ef3b4cb665bfa43c37f126a33f86dcdf34328c8c: Status 404 returned error can't find the container with id 29e185e0d944ef3124a2b576ef3b4cb665bfa43c37f126a33f86dcdf34328c8c Nov 03 21:15:28.606004 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:28.605971 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:15:28.606098 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:28.606035 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:15:28.606098 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:28.606078 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:15:28.607131 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:28.607096 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:15:28.607222 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:28.607157 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:15:28.607222 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:28.607190 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:15:28.897156 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:28.897116 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c2d98e6-472e-40bc-80df-efded30cbb5d" path="/var/lib/kubelet/pods/5c2d98e6-472e-40bc-80df-efded30cbb5d/volumes" Nov 03 21:15:28.897799 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:28.897773 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae494fad-b280-451c-8669-2340a689bec8" path="/var/lib/kubelet/pods/ae494fad-b280-451c-8669-2340a689bec8/volumes" Nov 03 21:15:29.263524 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:29.260980 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7" event={"ID":"308949ce-8a8c-4cd7-ac5a-a778aeaa4caf","Type":"ContainerStarted","Data":"43e8562ff98620ed23e061e5ff8dc920a1c71fe5fdfee982176edcbb9e2dbaec"} Nov 03 21:15:29.263524 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:29.261064 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7" event={"ID":"308949ce-8a8c-4cd7-ac5a-a778aeaa4caf","Type":"ContainerStarted","Data":"29e185e0d944ef3124a2b576ef3b4cb665bfa43c37f126a33f86dcdf34328c8c"} Nov 03 21:15:29.278459 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:29.266601 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz" event={"ID":"9f872846-a625-48d9-af4e-5854d31069ab","Type":"ContainerStarted","Data":"01fda3a0587b9907958119ada5eb8ebb342c590344b3bc92a3a89adfac74de54"} Nov 03 21:15:29.278459 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:29.266686 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz" event={"ID":"9f872846-a625-48d9-af4e-5854d31069ab","Type":"ContainerStarted","Data":"07314be4527e85add65454e85cc91e54886cdd35bb8d7f28cfe9ec6b37067d03"} Nov 03 21:15:29.300320 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:29.300277 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n" event={"ID":"5e88a59e-f72d-4345-8fc1-5c7113dc4638","Type":"ContainerStarted","Data":"ca186049766bc2caec16e2573a2a016318a5801ea41ace28bf5de4ba6a851f54"} Nov 03 21:15:29.300685 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:29.300645 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n" event={"ID":"5e88a59e-f72d-4345-8fc1-5c7113dc4638","Type":"ContainerStarted","Data":"23add24fae3a2610d1fba019089b46af5cfb8501ee811c3b0bcfe246177e9d99"} Nov 03 21:15:29.543959 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:29.542569 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7" podStartSLOduration=2.542550164 podStartE2EDuration="2.542550164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:29.411090317 +0000 UTC m=+2164.636410133" watchObservedRunningTime="2026-11-03 21:15:29.542550164 +0000 UTC m=+2164.767869960" Nov 03 21:15:29.543959 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:29.542804 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz" podStartSLOduration=2.5427982890000003 podStartE2EDuration="2.542798289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:29.539674039 +0000 UTC m=+2164.764993829" watchObservedRunningTime="2026-11-03 21:15:29.542798289 +0000 UTC m=+2164.768118085" Nov 03 21:15:29.679458 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:29.677399 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n" podStartSLOduration=2.6773778139999997 podStartE2EDuration="2.677377814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:29.676789151 +0000 UTC m=+2164.902108942" watchObservedRunningTime="2026-11-03 21:15:29.677377814 +0000 UTC m=+2164.902697607" Nov 03 21:15:30.115464 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:30.115133 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz" Nov 03 21:15:30.121465 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:30.120555 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Nov 03 21:15:30.121465 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:30.120617 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz" podUID="9f872846-a625-48d9-af4e-5854d31069ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Nov 03 21:15:30.159461 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:30.155590 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n" Nov 03 21:15:30.192459 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:30.179143 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body= Nov 03 21:15:30.192459 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:30.179217 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n" podUID="5e88a59e-f72d-4345-8fc1-5c7113dc4638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" Nov 03 21:15:30.192459 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:30.179491 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7" Nov 03 21:15:30.192459 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:30.185569 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body= Nov 03 21:15:30.192459 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:30.185626 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7" podUID="308949ce-8a8c-4cd7-ac5a-a778aeaa4caf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" Nov 03 21:15:31.104460 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:31.103885 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Nov 03 21:15:31.104460 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:31.103961 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz" podUID="9f872846-a625-48d9-af4e-5854d31069ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Nov 03 21:15:31.159461 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:31.155878 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body= Nov 03 21:15:31.159461 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:31.155953 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n" podUID="5e88a59e-f72d-4345-8fc1-5c7113dc4638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" Nov 03 21:15:31.164581 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:31.164541 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body= Nov 03 21:15:31.164862 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:31.164832 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7" podUID="308949ce-8a8c-4cd7-ac5a-a778aeaa4caf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" Nov 03 21:15:32.038217 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:32.038172 4305 scope.go:122] "RemoveContainer" containerID="464727cb6a9016bb6dba67b58aa975a8adc5acffd3bd585ff676b7c56a3388f8" Nov 03 21:15:32.048153 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:32.048133 4305 scope.go:122] "RemoveContainer" containerID="4cfb4c063c1bbbe62e1ca1fc0192559c0707b52c482cb306758bd0397f8f5359" Nov 03 21:15:32.114459 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:32.106902 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Nov 03 21:15:32.114459 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:32.106967 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz" podUID="9f872846-a625-48d9-af4e-5854d31069ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Nov 03 21:15:32.118924 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:32.118159 4305 scope.go:122] "RemoveContainer" containerID="0a8c351b3f7b2fd30e038ca961d55d967299ae906ecc77687ba09a2d3804c15d" Nov 03 21:15:32.138019 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:32.137995 4305 scope.go:122] "RemoveContainer" containerID="eccac510d12dbcac88497133af76a034cffcd5a96cb39a51281ea3e9e759515a" Nov 03 21:15:32.166471 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:32.164138 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body= Nov 03 21:15:32.166471 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:32.164198 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n" podUID="5e88a59e-f72d-4345-8fc1-5c7113dc4638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" Nov 03 21:15:32.166471 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:32.164276 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body= Nov 03 21:15:32.166471 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:32.164300 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7" podUID="308949ce-8a8c-4cd7-ac5a-a778aeaa4caf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" Nov 03 21:15:33.122461 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:33.121978 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Nov 03 21:15:33.122461 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:33.122053 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz" podUID="9f872846-a625-48d9-af4e-5854d31069ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Nov 03 21:15:33.170465 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:33.155964 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body= Nov 03 21:15:33.170465 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:33.156032 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n" podUID="5e88a59e-f72d-4345-8fc1-5c7113dc4638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" Nov 03 21:15:33.170465 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:33.164947 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body= Nov 03 21:15:33.170465 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:33.165007 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7" podUID="308949ce-8a8c-4cd7-ac5a-a778aeaa4caf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" Nov 03 21:15:34.119460 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:34.118469 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Nov 03 21:15:34.119460 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:34.118530 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz" podUID="9f872846-a625-48d9-af4e-5854d31069ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Nov 03 21:15:34.177459 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:34.177255 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body= Nov 03 21:15:34.177459 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:34.177319 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7" podUID="308949ce-8a8c-4cd7-ac5a-a778aeaa4caf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" Nov 03 21:15:34.178101 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:34.177914 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body= Nov 03 21:15:34.178101 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:34.177956 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n" podUID="5e88a59e-f72d-4345-8fc1-5c7113dc4638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" Nov 03 21:15:35.104764 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:35.104723 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Nov 03 21:15:35.104978 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:35.104795 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz" podUID="9f872846-a625-48d9-af4e-5854d31069ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Nov 03 21:15:35.156549 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:35.156510 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body= Nov 03 21:15:35.156773 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:35.156567 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n" podUID="5e88a59e-f72d-4345-8fc1-5c7113dc4638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" Nov 03 21:15:35.163042 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:35.162994 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body= Nov 03 21:15:35.163251 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:35.163054 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7" podUID="308949ce-8a8c-4cd7-ac5a-a778aeaa4caf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" Nov 03 21:15:36.104587 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:36.104533 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Nov 03 21:15:36.105142 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:36.104608 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz" podUID="9f872846-a625-48d9-af4e-5854d31069ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Nov 03 21:15:36.156287 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:36.156242 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body= Nov 03 21:15:36.156532 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:36.156314 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n" podUID="5e88a59e-f72d-4345-8fc1-5c7113dc4638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" Nov 03 21:15:36.163008 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:36.162975 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body= Nov 03 21:15:36.163170 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:36.163036 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7" podUID="308949ce-8a8c-4cd7-ac5a-a778aeaa4caf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" Nov 03 21:15:37.430247 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:37.430216 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz" Nov 03 21:15:37.430965 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:37.430938 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz" Nov 03 21:15:37.432448 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:37.432405 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz" Nov 03 21:15:37.598053 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:37.598016 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n" Nov 03 21:15:37.598489 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:37.598457 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n" Nov 03 21:15:37.599601 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:37.599576 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n" Nov 03 21:15:37.761983 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:37.761899 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7" Nov 03 21:15:37.762317 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:37.762291 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7" Nov 03 21:15:37.763117 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:15:37.763099 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7" Nov 03 21:15:46.321537 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:15:46.321499 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:16:09.508447 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:16:09.508341 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:16:16.320857 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:16:16.320809 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:16:30.990421 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:16:30.990385 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:16:32.386460 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:16:32.374701 4305 scope.go:122] "RemoveContainer" containerID="138c464707ad1a467a5950e4214fa58eeaaa5919a4ef1e2696f57c08bcb60ccd" Nov 03 21:16:32.428858 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:16:32.428831 4305 scope.go:122] "RemoveContainer" containerID="9be7384cebb7d0b8e1a43101e103e7f8c570fa09e6e0813a0cf3f6a8e915199d" Nov 03 21:16:32.438229 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:16:32.438202 4305 scope.go:122] "RemoveContainer" containerID="3ff1b9893ad058ade0ef7e8fd58b613703fd0d7835d9d13ce9f7fce445abaaeb" Nov 03 21:16:32.447744 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:16:32.447719 4305 scope.go:122] "RemoveContainer" containerID="759b16b59e1faab5e6782884565e483c016e3b3b41a8b68c131ba3753a7fa2d7" Nov 03 21:16:38.480458 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:16:38.476539 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:16:38.480458 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:16:38.476624 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz" podUID="9f872846-a625-48d9-af4e-5854d31069ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:16:38.643637 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:16:38.643595 4305 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:16:38.643845 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:16:38.643662 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n" podUID="5e88a59e-f72d-4345-8fc1-5c7113dc4638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:16:42.068878 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:16:42.068841 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:16:46.321932 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:16:46.321894 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:17:02.312393 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:17:02.312336 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:17:16.323892 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:17:16.323853 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:17:19.299492 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:17:19.299447 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:17:32.548341 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:17:32.548309 4305 scope.go:122] "RemoveContainer" containerID="f7634e00dc13bd882cc5d5fe561e6dedff6f8ff3cfa6ad82d25a69c144ab1efb" Nov 03 21:17:32.558185 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:17:32.558160 4305 scope.go:122] "RemoveContainer" containerID="e8d5b09aeadb497765f2889e8a7c6a7abfe40a8d6dab0921340fd82d1a3f8202" Nov 03 21:17:46.322193 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:17:46.322101 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:17:50.205373 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:17:50.205334 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:18:00.773408 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:00.773366 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:18:02.058240 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:02.058201 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0944521f-8t75q"] Nov 03 21:18:02.106748 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:02.106703 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07550c72-52hqf"] Nov 03 21:18:02.108792 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:02.108770 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0944521f-8t75q"] Nov 03 21:18:02.110565 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:02.110541 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07550c72-52hqf"] Nov 03 21:18:02.894550 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:02.894518 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5448207d-7597-4ebd-ad24-913b6564f5ad" path="/var/lib/kubelet/pods/5448207d-7597-4ebd-ad24-913b6564f5ad/volumes" Nov 03 21:18:02.895024 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:02.895008 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a5f2262-0ce7-4ad7-89f0-a1145d1da1dd" path="/var/lib/kubelet/pods/8a5f2262-0ce7-4ad7-89f0-a1145d1da1dd/volumes" Nov 03 21:18:13.320162 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:13.320119 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:18:16.322766 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:18:16.322720 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:18:20.590453 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:20.590410 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nw9m9"] Nov 03 21:18:20.590972 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:20.590635 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nw9m9" podUID="929bc9f6-7a97-49d1-b866-4d6db686b0b1" containerName="discovery" containerID="cri-o://9760815d3cf7de0d9f1865b22bab6c40dfc66a18ef96c384526a8c5b5246f5a6" gracePeriod=30 Nov 03 21:18:20.857992 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:20.857965 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nw9m9" Nov 03 21:18:20.951028 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:20.950994 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/929bc9f6-7a97-49d1-b866-4d6db686b0b1-istio-token\" (UniqueName: \"kubernetes.io/projected/929bc9f6-7a97-49d1-b866-4d6db686b0b1-istio-token\") pod \"929bc9f6-7a97-49d1-b866-4d6db686b0b1\" (UID: \"929bc9f6-7a97-49d1-b866-4d6db686b0b1\") " Nov 03 21:18:20.951212 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:20.951064 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/929bc9f6-7a97-49d1-b866-4d6db686b0b1-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/929bc9f6-7a97-49d1-b866-4d6db686b0b1-local-certs\") pod \"929bc9f6-7a97-49d1-b866-4d6db686b0b1\" (UID: \"929bc9f6-7a97-49d1-b866-4d6db686b0b1\") " Nov 03 21:18:20.951212 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:20.951161 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/929bc9f6-7a97-49d1-b866-4d6db686b0b1-cacerts\" (UniqueName: \"kubernetes.io/secret/929bc9f6-7a97-49d1-b866-4d6db686b0b1-cacerts\") pod \"929bc9f6-7a97-49d1-b866-4d6db686b0b1\" (UID: \"929bc9f6-7a97-49d1-b866-4d6db686b0b1\") " Nov 03 21:18:20.951316 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:20.951219 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/929bc9f6-7a97-49d1-b866-4d6db686b0b1-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/929bc9f6-7a97-49d1-b866-4d6db686b0b1-istio-csr-dns-cert\") pod \"929bc9f6-7a97-49d1-b866-4d6db686b0b1\" (UID: \"929bc9f6-7a97-49d1-b866-4d6db686b0b1\") " Nov 03 21:18:20.951316 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:20.951254 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/929bc9f6-7a97-49d1-b866-4d6db686b0b1-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/929bc9f6-7a97-49d1-b866-4d6db686b0b1-istio-kubeconfig\") pod \"929bc9f6-7a97-49d1-b866-4d6db686b0b1\" (UID: \"929bc9f6-7a97-49d1-b866-4d6db686b0b1\") " Nov 03 21:18:20.951316 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:20.951306 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/929bc9f6-7a97-49d1-b866-4d6db686b0b1-kube-api-access-9qb8v\" (UniqueName: \"kubernetes.io/projected/929bc9f6-7a97-49d1-b866-4d6db686b0b1-kube-api-access-9qb8v\") pod \"929bc9f6-7a97-49d1-b866-4d6db686b0b1\" (UID: \"929bc9f6-7a97-49d1-b866-4d6db686b0b1\") " Nov 03 21:18:20.951515 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:20.951403 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/929bc9f6-7a97-49d1-b866-4d6db686b0b1-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/929bc9f6-7a97-49d1-b866-4d6db686b0b1-istio-csr-ca-configmap\") pod \"929bc9f6-7a97-49d1-b866-4d6db686b0b1\" (UID: \"929bc9f6-7a97-49d1-b866-4d6db686b0b1\") " Nov 03 21:18:20.952423 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:20.952384 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/929bc9f6-7a97-49d1-b866-4d6db686b0b1-istio-csr-ca-configmap" pod "929bc9f6-7a97-49d1-b866-4d6db686b0b1" (UID: "929bc9f6-7a97-49d1-b866-4d6db686b0b1"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:18:20.953767 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:20.953732 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/929bc9f6-7a97-49d1-b866-4d6db686b0b1-istio-token" pod "929bc9f6-7a97-49d1-b866-4d6db686b0b1" (UID: "929bc9f6-7a97-49d1-b866-4d6db686b0b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:18:20.953900 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:20.953804 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/929bc9f6-7a97-49d1-b866-4d6db686b0b1-kube-api-access-9qb8v" pod "929bc9f6-7a97-49d1-b866-4d6db686b0b1" (UID: "929bc9f6-7a97-49d1-b866-4d6db686b0b1"). InnerVolumeSpecName "kube-api-access-9qb8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:18:20.954001 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:20.953981 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/929bc9f6-7a97-49d1-b866-4d6db686b0b1-local-certs" pod "929bc9f6-7a97-49d1-b866-4d6db686b0b1" (UID: "929bc9f6-7a97-49d1-b866-4d6db686b0b1"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:18:20.954001 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:20.953988 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/929bc9f6-7a97-49d1-b866-4d6db686b0b1-cacerts" pod "929bc9f6-7a97-49d1-b866-4d6db686b0b1" (UID: "929bc9f6-7a97-49d1-b866-4d6db686b0b1"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:18:20.954120 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:20.954103 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/929bc9f6-7a97-49d1-b866-4d6db686b0b1-istio-csr-dns-cert" pod "929bc9f6-7a97-49d1-b866-4d6db686b0b1" (UID: "929bc9f6-7a97-49d1-b866-4d6db686b0b1"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:18:20.954163 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:20.954148 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/929bc9f6-7a97-49d1-b866-4d6db686b0b1-istio-kubeconfig" pod "929bc9f6-7a97-49d1-b866-4d6db686b0b1" (UID: "929bc9f6-7a97-49d1-b866-4d6db686b0b1"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:18:21.053367 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:21.053269 4305 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/929bc9f6-7a97-49d1-b866-4d6db686b0b1-istio-csr-ca-configmap\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:18:21.053367 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:21.053354 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/929bc9f6-7a97-49d1-b866-4d6db686b0b1-istio-token\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:18:21.053709 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:21.053380 4305 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/929bc9f6-7a97-49d1-b866-4d6db686b0b1-local-certs\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:18:21.053709 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:21.053394 4305 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/929bc9f6-7a97-49d1-b866-4d6db686b0b1-cacerts\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:18:21.053709 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:21.053408 4305 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/929bc9f6-7a97-49d1-b866-4d6db686b0b1-istio-csr-dns-cert\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:18:21.053709 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:21.053420 4305 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/929bc9f6-7a97-49d1-b866-4d6db686b0b1-istio-kubeconfig\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:18:21.053709 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:21.053449 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qb8v\" (UniqueName: \"kubernetes.io/projected/929bc9f6-7a97-49d1-b866-4d6db686b0b1-kube-api-access-9qb8v\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:18:21.491841 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:21.491805 4305 generic.go:356] "Generic (PLEG): container finished" podID="929bc9f6-7a97-49d1-b866-4d6db686b0b1" containerID="9760815d3cf7de0d9f1865b22bab6c40dfc66a18ef96c384526a8c5b5246f5a6" exitCode=0 Nov 03 21:18:21.492180 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:21.491861 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nw9m9" Nov 03 21:18:21.492180 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:21.491893 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nw9m9" event={"ID":"929bc9f6-7a97-49d1-b866-4d6db686b0b1","Type":"ContainerDied","Data":"9760815d3cf7de0d9f1865b22bab6c40dfc66a18ef96c384526a8c5b5246f5a6"} Nov 03 21:18:21.492180 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:21.491955 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nw9m9" event={"ID":"929bc9f6-7a97-49d1-b866-4d6db686b0b1","Type":"ContainerDied","Data":"3b79bbb91fae6e8f2f6e2d57c147427c2f077590d8e48c5558de6421d3aa80fd"} Nov 03 21:18:21.492180 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:21.491980 4305 scope.go:122] "RemoveContainer" containerID="9760815d3cf7de0d9f1865b22bab6c40dfc66a18ef96c384526a8c5b5246f5a6" Nov 03 21:18:21.503042 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:21.503022 4305 scope.go:122] "RemoveContainer" containerID="9760815d3cf7de0d9f1865b22bab6c40dfc66a18ef96c384526a8c5b5246f5a6" Nov 03 21:18:21.503370 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:18:21.503346 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9760815d3cf7de0d9f1865b22bab6c40dfc66a18ef96c384526a8c5b5246f5a6\": container with ID starting with 9760815d3cf7de0d9f1865b22bab6c40dfc66a18ef96c384526a8c5b5246f5a6 not found: ID does not exist" containerID="9760815d3cf7de0d9f1865b22bab6c40dfc66a18ef96c384526a8c5b5246f5a6" Nov 03 21:18:21.503479 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:21.503379 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9760815d3cf7de0d9f1865b22bab6c40dfc66a18ef96c384526a8c5b5246f5a6"} err="failed to get container status \"9760815d3cf7de0d9f1865b22bab6c40dfc66a18ef96c384526a8c5b5246f5a6\": rpc error: code = NotFound desc = could not find container \"9760815d3cf7de0d9f1865b22bab6c40dfc66a18ef96c384526a8c5b5246f5a6\": container with ID starting with 9760815d3cf7de0d9f1865b22bab6c40dfc66a18ef96c384526a8c5b5246f5a6 not found: ID does not exist" Nov 03 21:18:21.659488 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:21.659453 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nw9m9"] Nov 03 21:18:21.708644 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:21.708609 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nw9m9"] Nov 03 21:18:22.894609 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:22.894572 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="929bc9f6-7a97-49d1-b866-4d6db686b0b1" path="/var/lib/kubelet/pods/929bc9f6-7a97-49d1-b866-4d6db686b0b1/volumes" Nov 03 21:18:23.411496 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:23.411455 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mlgs7"] Nov 03 21:18:23.411892 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:23.411879 4305 cpu_manager.go:414] "containerMap: removing container" podUID="929bc9f6-7a97-49d1-b866-4d6db686b0b1" containerName="discovery" Nov 03 21:18:23.411952 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:23.411893 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="929bc9f6-7a97-49d1-b866-4d6db686b0b1" containerName="discovery" Nov 03 21:18:23.412003 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:23.411994 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="929bc9f6-7a97-49d1-b866-4d6db686b0b1" containerName="discovery" Nov 03 21:18:23.416351 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:23.416335 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mlgs7" Nov 03 21:18:23.521580 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:23.521535 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:18:23.521777 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:23.521535 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77vtd\"" Nov 03 21:18:23.521777 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:23.521559 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:18:23.521777 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:23.521559 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:18:23.524011 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:23.523988 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mlgs7"] Nov 03 21:18:23.580042 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:23.580008 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcfz5\" (UniqueName: \"kubernetes.io/projected/ef0ad8a6-0850-46fe-8e2b-5069bf15835a-kube-api-access-zcfz5\") pod \"maas-api-revoke-sub-ae594979-mlgs7\" (UID: \"ef0ad8a6-0850-46fe-8e2b-5069bf15835a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mlgs7" Nov 03 21:18:23.580202 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:23.580064 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef0ad8a6-0850-46fe-8e2b-5069bf15835a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-mlgs7\" (UID: \"ef0ad8a6-0850-46fe-8e2b-5069bf15835a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mlgs7" Nov 03 21:18:23.580202 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:23.580164 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef0ad8a6-0850-46fe-8e2b-5069bf15835a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-mlgs7\" (UID: \"ef0ad8a6-0850-46fe-8e2b-5069bf15835a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mlgs7" Nov 03 21:18:23.680921 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:23.680829 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcfz5\" (UniqueName: \"kubernetes.io/projected/ef0ad8a6-0850-46fe-8e2b-5069bf15835a-kube-api-access-zcfz5\") pod \"maas-api-revoke-sub-ae594979-mlgs7\" (UID: \"ef0ad8a6-0850-46fe-8e2b-5069bf15835a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mlgs7" Nov 03 21:18:23.680921 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:23.680878 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef0ad8a6-0850-46fe-8e2b-5069bf15835a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-mlgs7\" (UID: \"ef0ad8a6-0850-46fe-8e2b-5069bf15835a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mlgs7" Nov 03 21:18:23.680921 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:23.680912 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef0ad8a6-0850-46fe-8e2b-5069bf15835a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-mlgs7\" (UID: \"ef0ad8a6-0850-46fe-8e2b-5069bf15835a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mlgs7" Nov 03 21:18:23.681559 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:23.681532 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef0ad8a6-0850-46fe-8e2b-5069bf15835a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-mlgs7\" (UID: \"ef0ad8a6-0850-46fe-8e2b-5069bf15835a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mlgs7" Nov 03 21:18:23.683601 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:23.683570 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef0ad8a6-0850-46fe-8e2b-5069bf15835a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-mlgs7\" (UID: \"ef0ad8a6-0850-46fe-8e2b-5069bf15835a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mlgs7" Nov 03 21:18:23.742017 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:23.741980 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcfz5\" (UniqueName: \"kubernetes.io/projected/ef0ad8a6-0850-46fe-8e2b-5069bf15835a-kube-api-access-zcfz5\") pod \"maas-api-revoke-sub-ae594979-mlgs7\" (UID: \"ef0ad8a6-0850-46fe-8e2b-5069bf15835a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mlgs7" Nov 03 21:18:24.029815 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:24.029713 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mlgs7" Nov 03 21:18:24.408897 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:24.408868 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mlgs7"] Nov 03 21:18:24.410863 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:18:24.410834 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef0ad8a6_0850_46fe_8e2b_5069bf15835a.slice/crio-5bce5fe42c8bff92ceac9487eeb623539aef7a88e62cd7eab0dc368d9fcdcf35 WatchSource:0}: Error finding container 5bce5fe42c8bff92ceac9487eeb623539aef7a88e62cd7eab0dc368d9fcdcf35: Status 404 returned error can't find the container with id 5bce5fe42c8bff92ceac9487eeb623539aef7a88e62cd7eab0dc368d9fcdcf35 Nov 03 21:18:24.512379 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:24.512343 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mlgs7" event={"ID":"ef0ad8a6-0850-46fe-8e2b-5069bf15835a","Type":"ContainerStarted","Data":"5bce5fe42c8bff92ceac9487eeb623539aef7a88e62cd7eab0dc368d9fcdcf35"} Nov 03 21:18:25.520713 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:25.520675 4305 generic.go:356] "Generic (PLEG): container finished" podID="ef0ad8a6-0850-46fe-8e2b-5069bf15835a" containerID="1759a8098b58453253ed8acd4b1fd82577ff6215780cf8b7fb619f7e79ee54a7" exitCode=0 Nov 03 21:18:25.521188 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:25.520763 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mlgs7" event={"ID":"ef0ad8a6-0850-46fe-8e2b-5069bf15835a","Type":"ContainerDied","Data":"1759a8098b58453253ed8acd4b1fd82577ff6215780cf8b7fb619f7e79ee54a7"} Nov 03 21:18:26.675404 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:26.675378 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mlgs7" Nov 03 21:18:26.813607 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:26.813510 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef0ad8a6-0850-46fe-8e2b-5069bf15835a-kube-api-access-zcfz5\" (UniqueName: \"kubernetes.io/projected/ef0ad8a6-0850-46fe-8e2b-5069bf15835a-kube-api-access-zcfz5\") pod \"ef0ad8a6-0850-46fe-8e2b-5069bf15835a\" (UID: \"ef0ad8a6-0850-46fe-8e2b-5069bf15835a\") " Nov 03 21:18:26.813798 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:26.813737 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef0ad8a6-0850-46fe-8e2b-5069bf15835a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef0ad8a6-0850-46fe-8e2b-5069bf15835a-maas-api-openshift-ca\") pod \"ef0ad8a6-0850-46fe-8e2b-5069bf15835a\" (UID: \"ef0ad8a6-0850-46fe-8e2b-5069bf15835a\") " Nov 03 21:18:26.813798 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:26.813776 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef0ad8a6-0850-46fe-8e2b-5069bf15835a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef0ad8a6-0850-46fe-8e2b-5069bf15835a-maas-api-xks-ca\") pod \"ef0ad8a6-0850-46fe-8e2b-5069bf15835a\" (UID: \"ef0ad8a6-0850-46fe-8e2b-5069bf15835a\") " Nov 03 21:18:26.814155 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:26.814114 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef0ad8a6-0850-46fe-8e2b-5069bf15835a-maas-api-openshift-ca" pod "ef0ad8a6-0850-46fe-8e2b-5069bf15835a" (UID: "ef0ad8a6-0850-46fe-8e2b-5069bf15835a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:18:26.816075 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:26.816054 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef0ad8a6-0850-46fe-8e2b-5069bf15835a-maas-api-xks-ca" pod "ef0ad8a6-0850-46fe-8e2b-5069bf15835a" (UID: "ef0ad8a6-0850-46fe-8e2b-5069bf15835a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:18:26.816349 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:26.816330 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef0ad8a6-0850-46fe-8e2b-5069bf15835a-kube-api-access-zcfz5" pod "ef0ad8a6-0850-46fe-8e2b-5069bf15835a" (UID: "ef0ad8a6-0850-46fe-8e2b-5069bf15835a"). InnerVolumeSpecName "kube-api-access-zcfz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:18:26.914888 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:26.914856 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef0ad8a6-0850-46fe-8e2b-5069bf15835a-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:18:26.914888 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:26.914887 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef0ad8a6-0850-46fe-8e2b-5069bf15835a-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:18:26.915089 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:26.914900 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcfz5\" (UniqueName: \"kubernetes.io/projected/ef0ad8a6-0850-46fe-8e2b-5069bf15835a-kube-api-access-zcfz5\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:18:27.533838 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:27.533799 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mlgs7" Nov 03 21:18:27.534024 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:27.533837 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mlgs7" event={"ID":"ef0ad8a6-0850-46fe-8e2b-5069bf15835a","Type":"ContainerDied","Data":"5bce5fe42c8bff92ceac9487eeb623539aef7a88e62cd7eab0dc368d9fcdcf35"} Nov 03 21:18:27.534024 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:27.533923 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5bce5fe42c8bff92ceac9487eeb623539aef7a88e62cd7eab0dc368d9fcdcf35" Nov 03 21:18:36.306237 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:36.306200 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:18:45.889724 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:45.889679 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:18:46.320847 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:18:46.320810 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:18:51.503419 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:51.503378 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-smcsd"] Nov 03 21:18:51.503895 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:51.503880 4305 cpu_manager.go:414] "containerMap: removing container" podUID="ef0ad8a6-0850-46fe-8e2b-5069bf15835a" containerName="revoke-keys" Nov 03 21:18:51.503951 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:51.503896 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef0ad8a6-0850-46fe-8e2b-5069bf15835a" containerName="revoke-keys" Nov 03 21:18:51.503989 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:51.503970 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef0ad8a6-0850-46fe-8e2b-5069bf15835a" containerName="revoke-keys" Nov 03 21:18:51.507133 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:51.507116 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-smcsd" Nov 03 21:18:51.610797 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:51.610766 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:18:51.610797 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:51.610766 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:18:51.611018 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:51.610811 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77vtd\"" Nov 03 21:18:51.611018 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:51.610869 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:18:51.613472 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:51.613447 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-smcsd"] Nov 03 21:18:51.613883 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:51.613863 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f08548e-3650-4b24-84e9-1ae1403ab828-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-smcsd\" (UID: \"8f08548e-3650-4b24-84e9-1ae1403ab828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-smcsd" Nov 03 21:18:51.613940 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:51.613928 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqthr\" (UniqueName: \"kubernetes.io/projected/8f08548e-3650-4b24-84e9-1ae1403ab828-kube-api-access-pqthr\") pod \"maas-api-revoke-sub-a4aef7d7-smcsd\" (UID: \"8f08548e-3650-4b24-84e9-1ae1403ab828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-smcsd" Nov 03 21:18:51.614020 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:51.614006 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f08548e-3650-4b24-84e9-1ae1403ab828-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-smcsd\" (UID: \"8f08548e-3650-4b24-84e9-1ae1403ab828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-smcsd" Nov 03 21:18:51.715404 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:51.715360 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f08548e-3650-4b24-84e9-1ae1403ab828-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-smcsd\" (UID: \"8f08548e-3650-4b24-84e9-1ae1403ab828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-smcsd" Nov 03 21:18:51.715655 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:51.715482 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqthr\" (UniqueName: \"kubernetes.io/projected/8f08548e-3650-4b24-84e9-1ae1403ab828-kube-api-access-pqthr\") pod \"maas-api-revoke-sub-a4aef7d7-smcsd\" (UID: \"8f08548e-3650-4b24-84e9-1ae1403ab828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-smcsd" Nov 03 21:18:51.715655 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:51.715502 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f08548e-3650-4b24-84e9-1ae1403ab828-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-smcsd\" (UID: \"8f08548e-3650-4b24-84e9-1ae1403ab828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-smcsd" Nov 03 21:18:51.716187 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:51.716168 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f08548e-3650-4b24-84e9-1ae1403ab828-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-smcsd\" (UID: \"8f08548e-3650-4b24-84e9-1ae1403ab828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-smcsd" Nov 03 21:18:51.717943 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:51.717919 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f08548e-3650-4b24-84e9-1ae1403ab828-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-smcsd\" (UID: \"8f08548e-3650-4b24-84e9-1ae1403ab828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-smcsd" Nov 03 21:18:51.776955 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:51.776882 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqthr\" (UniqueName: \"kubernetes.io/projected/8f08548e-3650-4b24-84e9-1ae1403ab828-kube-api-access-pqthr\") pod \"maas-api-revoke-sub-a4aef7d7-smcsd\" (UID: \"8f08548e-3650-4b24-84e9-1ae1403ab828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-smcsd" Nov 03 21:18:51.820809 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:51.820774 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-smcsd" Nov 03 21:18:52.194559 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:52.194533 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-smcsd"] Nov 03 21:18:52.196893 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:18:52.196860 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f08548e_3650_4b24_84e9_1ae1403ab828.slice/crio-1c698ed877f9f1dc53f2284ddc8529f8bfd1f54e6953066b5cd8ae256ea43aa9 WatchSource:0}: Error finding container 1c698ed877f9f1dc53f2284ddc8529f8bfd1f54e6953066b5cd8ae256ea43aa9: Status 404 returned error can't find the container with id 1c698ed877f9f1dc53f2284ddc8529f8bfd1f54e6953066b5cd8ae256ea43aa9 Nov 03 21:18:52.711875 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:52.711772 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-smcsd" event={"ID":"8f08548e-3650-4b24-84e9-1ae1403ab828","Type":"ContainerStarted","Data":"302786a7b11fe2208c90284f84a79fb881bd52fa24194fde3ada952877629117"} Nov 03 21:18:52.711875 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:52.711868 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-smcsd" event={"ID":"8f08548e-3650-4b24-84e9-1ae1403ab828","Type":"ContainerStarted","Data":"1c698ed877f9f1dc53f2284ddc8529f8bfd1f54e6953066b5cd8ae256ea43aa9"} Nov 03 21:18:52.828667 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:52.828603 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-smcsd" podStartSLOduration=1.828585773 podStartE2EDuration="1.828585773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:18:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:18:52.827220534 +0000 UTC m=+2368.052540349" watchObservedRunningTime="2026-11-03 21:18:52.828585773 +0000 UTC m=+2368.053905565" Nov 03 21:18:53.718428 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:53.718395 4305 generic.go:356] "Generic (PLEG): container finished" podID="8f08548e-3650-4b24-84e9-1ae1403ab828" containerID="302786a7b11fe2208c90284f84a79fb881bd52fa24194fde3ada952877629117" exitCode=0 Nov 03 21:18:53.718913 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:53.718485 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-smcsd" event={"ID":"8f08548e-3650-4b24-84e9-1ae1403ab828","Type":"ContainerDied","Data":"302786a7b11fe2208c90284f84a79fb881bd52fa24194fde3ada952877629117"} Nov 03 21:18:54.868802 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:54.868774 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-smcsd" Nov 03 21:18:54.945798 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:54.945764 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f08548e-3650-4b24-84e9-1ae1403ab828-kube-api-access-pqthr\" (UniqueName: \"kubernetes.io/projected/8f08548e-3650-4b24-84e9-1ae1403ab828-kube-api-access-pqthr\") pod \"8f08548e-3650-4b24-84e9-1ae1403ab828\" (UID: \"8f08548e-3650-4b24-84e9-1ae1403ab828\") " Nov 03 21:18:54.945999 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:54.945951 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f08548e-3650-4b24-84e9-1ae1403ab828-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f08548e-3650-4b24-84e9-1ae1403ab828-maas-api-openshift-ca\") pod \"8f08548e-3650-4b24-84e9-1ae1403ab828\" (UID: \"8f08548e-3650-4b24-84e9-1ae1403ab828\") " Nov 03 21:18:54.946095 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:54.946001 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f08548e-3650-4b24-84e9-1ae1403ab828-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f08548e-3650-4b24-84e9-1ae1403ab828-maas-api-xks-ca\") pod \"8f08548e-3650-4b24-84e9-1ae1403ab828\" (UID: \"8f08548e-3650-4b24-84e9-1ae1403ab828\") " Nov 03 21:18:54.946328 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:54.946303 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f08548e-3650-4b24-84e9-1ae1403ab828-maas-api-openshift-ca" pod "8f08548e-3650-4b24-84e9-1ae1403ab828" (UID: "8f08548e-3650-4b24-84e9-1ae1403ab828"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:18:54.948107 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:54.948088 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f08548e-3650-4b24-84e9-1ae1403ab828-kube-api-access-pqthr" pod "8f08548e-3650-4b24-84e9-1ae1403ab828" (UID: "8f08548e-3650-4b24-84e9-1ae1403ab828"). InnerVolumeSpecName "kube-api-access-pqthr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:18:54.948186 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:54.948107 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f08548e-3650-4b24-84e9-1ae1403ab828-maas-api-xks-ca" pod "8f08548e-3650-4b24-84e9-1ae1403ab828" (UID: "8f08548e-3650-4b24-84e9-1ae1403ab828"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:18:55.047805 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:55.047722 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f08548e-3650-4b24-84e9-1ae1403ab828-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:18:55.047805 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:55.047750 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f08548e-3650-4b24-84e9-1ae1403ab828-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:18:55.047805 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:55.047760 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqthr\" (UniqueName: \"kubernetes.io/projected/8f08548e-3650-4b24-84e9-1ae1403ab828-kube-api-access-pqthr\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:18:55.730609 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:55.730576 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-smcsd" event={"ID":"8f08548e-3650-4b24-84e9-1ae1403ab828","Type":"ContainerDied","Data":"1c698ed877f9f1dc53f2284ddc8529f8bfd1f54e6953066b5cd8ae256ea43aa9"} Nov 03 21:18:55.730609 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:55.730593 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-smcsd" Nov 03 21:18:55.730884 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:18:55.730623 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c698ed877f9f1dc53f2284ddc8529f8bfd1f54e6953066b5cd8ae256ea43aa9" Nov 03 21:19:04.393704 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:04.393662 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:19:07.053458 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:07.053348 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3a7b6c5-4rgn4"] Nov 03 21:19:07.103025 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:07.102991 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ae29793-n5kjl"] Nov 03 21:19:07.105159 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:07.105140 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3a7b6c5-4rgn4"] Nov 03 21:19:07.106846 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:07.106828 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ae29793-n5kjl"] Nov 03 21:19:08.895729 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:08.895695 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07f1475c-e840-466e-bc78-75665a1468ba" path="/var/lib/kubelet/pods/07f1475c-e840-466e-bc78-75665a1468ba/volumes" Nov 03 21:19:08.896165 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:08.896114 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de2f5dab-53d4-4e0d-be0e-e71aa193705a" path="/var/lib/kubelet/pods/de2f5dab-53d4-4e0d-be0e-e71aa193705a/volumes" Nov 03 21:19:12.131642 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:12.131605 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93d44e89-kp8g9"] Nov 03 21:19:12.185697 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:12.185663 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93d44e89-kp8g9"] Nov 03 21:19:12.920190 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:12.920135 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da5ca27e-9637-4eea-b462-06f298f367de" path="/var/lib/kubelet/pods/da5ca27e-9637-4eea-b462-06f298f367de/volumes" Nov 03 21:19:16.321414 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:19:16.321376 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:19:16.816655 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:16.816618 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:19:22.047972 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:22.047921 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adfa0d96-rqh4f"] Nov 03 21:19:22.097799 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:22.097764 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adfa0d96-rqh4f"] Nov 03 21:19:22.099624 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:22.099602 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vscbw"] Nov 03 21:19:22.100038 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:22.100026 4305 cpu_manager.go:414] "containerMap: removing container" podUID="8f08548e-3650-4b24-84e9-1ae1403ab828" containerName="revoke-keys" Nov 03 21:19:22.100089 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:22.100040 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f08548e-3650-4b24-84e9-1ae1403ab828" containerName="revoke-keys" Nov 03 21:19:22.100132 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:22.100123 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f08548e-3650-4b24-84e9-1ae1403ab828" containerName="revoke-keys" Nov 03 21:19:22.103231 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:22.103217 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vscbw" Nov 03 21:19:22.121881 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:22.121853 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f19c63e0-0d0c-4b63-a59f-7aca9057efc2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-vscbw\" (UID: \"f19c63e0-0d0c-4b63-a59f-7aca9057efc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vscbw" Nov 03 21:19:22.122014 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:22.121892 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phfkm\" (UniqueName: \"kubernetes.io/projected/f19c63e0-0d0c-4b63-a59f-7aca9057efc2-kube-api-access-phfkm\") pod \"maas-api-revoke-sub-79805ca9-vscbw\" (UID: \"f19c63e0-0d0c-4b63-a59f-7aca9057efc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vscbw" Nov 03 21:19:22.122014 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:22.121929 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f19c63e0-0d0c-4b63-a59f-7aca9057efc2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-vscbw\" (UID: \"f19c63e0-0d0c-4b63-a59f-7aca9057efc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vscbw" Nov 03 21:19:22.157945 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:22.157855 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:19:22.157945 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:22.157870 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:19:22.158188 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:22.158095 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:19:22.158283 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:22.158254 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77vtd\"" Nov 03 21:19:22.161007 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:22.160985 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vscbw"] Nov 03 21:19:22.223310 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:22.223278 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f19c63e0-0d0c-4b63-a59f-7aca9057efc2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-vscbw\" (UID: \"f19c63e0-0d0c-4b63-a59f-7aca9057efc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vscbw" Nov 03 21:19:22.223502 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:22.223330 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phfkm\" (UniqueName: \"kubernetes.io/projected/f19c63e0-0d0c-4b63-a59f-7aca9057efc2-kube-api-access-phfkm\") pod \"maas-api-revoke-sub-79805ca9-vscbw\" (UID: \"f19c63e0-0d0c-4b63-a59f-7aca9057efc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vscbw" Nov 03 21:19:22.223502 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:22.223384 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f19c63e0-0d0c-4b63-a59f-7aca9057efc2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-vscbw\" (UID: \"f19c63e0-0d0c-4b63-a59f-7aca9057efc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vscbw" Nov 03 21:19:22.224084 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:22.224063 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f19c63e0-0d0c-4b63-a59f-7aca9057efc2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-vscbw\" (UID: \"f19c63e0-0d0c-4b63-a59f-7aca9057efc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vscbw" Nov 03 21:19:22.225677 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:22.225655 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f19c63e0-0d0c-4b63-a59f-7aca9057efc2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-vscbw\" (UID: \"f19c63e0-0d0c-4b63-a59f-7aca9057efc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vscbw" Nov 03 21:19:22.284638 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:22.284608 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phfkm\" (UniqueName: \"kubernetes.io/projected/f19c63e0-0d0c-4b63-a59f-7aca9057efc2-kube-api-access-phfkm\") pod \"maas-api-revoke-sub-79805ca9-vscbw\" (UID: \"f19c63e0-0d0c-4b63-a59f-7aca9057efc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vscbw" Nov 03 21:19:22.415649 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:22.415612 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vscbw" Nov 03 21:19:22.791721 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:22.791695 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vscbw"] Nov 03 21:19:22.794007 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:19:22.793977 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf19c63e0_0d0c_4b63_a59f_7aca9057efc2.slice/crio-9829e0149d079fe860c2d91a051ac2081e21a29ebc6909902979d1571d0796c2 WatchSource:0}: Error finding container 9829e0149d079fe860c2d91a051ac2081e21a29ebc6909902979d1571d0796c2: Status 404 returned error can't find the container with id 9829e0149d079fe860c2d91a051ac2081e21a29ebc6909902979d1571d0796c2 Nov 03 21:19:22.893838 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:22.893801 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2e9589d-058b-47fb-a28d-9b92a855c9ca" path="/var/lib/kubelet/pods/a2e9589d-058b-47fb-a28d-9b92a855c9ca/volumes" Nov 03 21:19:22.894923 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:22.894898 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vscbw" event={"ID":"f19c63e0-0d0c-4b63-a59f-7aca9057efc2","Type":"ContainerStarted","Data":"9829e0149d079fe860c2d91a051ac2081e21a29ebc6909902979d1571d0796c2"} Nov 03 21:19:23.053385 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:23.053342 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-383e06ff-tzdkz"] Nov 03 21:19:23.055665 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:23.055641 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-383e06ff-tzdkz"] Nov 03 21:19:23.902181 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:23.902150 4305 generic.go:356] "Generic (PLEG): container finished" podID="f19c63e0-0d0c-4b63-a59f-7aca9057efc2" containerID="7afc971a7a4f25587ce7355d49e179f35aa85798431569a816987220b0ece5c9" exitCode=0 Nov 03 21:19:23.902384 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:23.902244 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vscbw" event={"ID":"f19c63e0-0d0c-4b63-a59f-7aca9057efc2","Type":"ContainerDied","Data":"7afc971a7a4f25587ce7355d49e179f35aa85798431569a816987220b0ece5c9"} Nov 03 21:19:24.894697 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:24.894664 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a20ce38b-39a2-4e14-92f9-ad37dd2ad124" path="/var/lib/kubelet/pods/a20ce38b-39a2-4e14-92f9-ad37dd2ad124/volumes" Nov 03 21:19:25.046468 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:25.046425 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vscbw" Nov 03 21:19:25.153237 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:25.153141 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f19c63e0-0d0c-4b63-a59f-7aca9057efc2-kube-api-access-phfkm\" (UniqueName: \"kubernetes.io/projected/f19c63e0-0d0c-4b63-a59f-7aca9057efc2-kube-api-access-phfkm\") pod \"f19c63e0-0d0c-4b63-a59f-7aca9057efc2\" (UID: \"f19c63e0-0d0c-4b63-a59f-7aca9057efc2\") " Nov 03 21:19:25.153237 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:25.153226 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f19c63e0-0d0c-4b63-a59f-7aca9057efc2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f19c63e0-0d0c-4b63-a59f-7aca9057efc2-maas-api-xks-ca\") pod \"f19c63e0-0d0c-4b63-a59f-7aca9057efc2\" (UID: \"f19c63e0-0d0c-4b63-a59f-7aca9057efc2\") " Nov 03 21:19:25.153534 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:25.153397 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f19c63e0-0d0c-4b63-a59f-7aca9057efc2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f19c63e0-0d0c-4b63-a59f-7aca9057efc2-maas-api-openshift-ca\") pod \"f19c63e0-0d0c-4b63-a59f-7aca9057efc2\" (UID: \"f19c63e0-0d0c-4b63-a59f-7aca9057efc2\") " Nov 03 21:19:25.153739 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:25.153713 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f19c63e0-0d0c-4b63-a59f-7aca9057efc2-maas-api-openshift-ca" pod "f19c63e0-0d0c-4b63-a59f-7aca9057efc2" (UID: "f19c63e0-0d0c-4b63-a59f-7aca9057efc2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:19:25.155530 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:25.155506 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f19c63e0-0d0c-4b63-a59f-7aca9057efc2-kube-api-access-phfkm" pod "f19c63e0-0d0c-4b63-a59f-7aca9057efc2" (UID: "f19c63e0-0d0c-4b63-a59f-7aca9057efc2"). InnerVolumeSpecName "kube-api-access-phfkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:19:25.155620 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:25.155537 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f19c63e0-0d0c-4b63-a59f-7aca9057efc2-maas-api-xks-ca" pod "f19c63e0-0d0c-4b63-a59f-7aca9057efc2" (UID: "f19c63e0-0d0c-4b63-a59f-7aca9057efc2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:19:25.254379 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:25.254341 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f19c63e0-0d0c-4b63-a59f-7aca9057efc2-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:19:25.254379 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:25.254368 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phfkm\" (UniqueName: \"kubernetes.io/projected/f19c63e0-0d0c-4b63-a59f-7aca9057efc2-kube-api-access-phfkm\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:19:25.254379 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:25.254382 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f19c63e0-0d0c-4b63-a59f-7aca9057efc2-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:19:25.913104 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:25.913077 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vscbw" Nov 03 21:19:25.913536 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:25.913104 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vscbw" event={"ID":"f19c63e0-0d0c-4b63-a59f-7aca9057efc2","Type":"ContainerDied","Data":"9829e0149d079fe860c2d91a051ac2081e21a29ebc6909902979d1571d0796c2"} Nov 03 21:19:25.913536 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:25.913151 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9829e0149d079fe860c2d91a051ac2081e21a29ebc6909902979d1571d0796c2" Nov 03 21:19:32.601356 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:32.601322 4305 scope.go:122] "RemoveContainer" containerID="8c8bcd4e3c09e5f9a6d17936d53669c10a0038153ee3029a1a6ef9259f6de8c9" Nov 03 21:19:32.610969 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:32.610948 4305 scope.go:122] "RemoveContainer" containerID="b631d629c3e97e14ca9b4ac44b9f2e22484db843dda7e6bb5831f19f07855524" Nov 03 21:19:32.619719 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:32.619694 4305 scope.go:122] "RemoveContainer" containerID="69651833ddc73ae7bcbe01903d98052cb0cb1248d93bc00d6688e69ae789248b" Nov 03 21:19:32.628416 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:32.628397 4305 scope.go:122] "RemoveContainer" containerID="22b644f0ac2be81397cd6fca321a384427619a498e5af71ae61eb84895f43561" Nov 03 21:19:32.637154 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:32.637130 4305 scope.go:122] "RemoveContainer" containerID="0ec7145920fb66e107dd59a3dcee6480190d33cf572d9619856ed24c712964bd" Nov 03 21:19:32.645448 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:32.645413 4305 scope.go:122] "RemoveContainer" containerID="372b7fa03e7ff0ba314086a410a630013329a2bb1896c82b760559d5ef3aaf0a" Nov 03 21:19:32.654133 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:32.654109 4305 scope.go:122] "RemoveContainer" containerID="4e25d72425f9f2dd7535c1b42e421decd8efe591593189de2b7ff301599a94f7" Nov 03 21:19:32.662762 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:32.662734 4305 scope.go:122] "RemoveContainer" containerID="56f8da079380bf114103cfbb00a7a3d91a2e4c69c5a822447d9d03b986dc4712" Nov 03 21:19:35.068333 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:35.068285 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:19:46.320389 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:19:46.320349 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:19:46.894466 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:46.894407 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:19:54.385974 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:54.385936 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-czgbn"] Nov 03 21:19:54.386476 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:54.386387 4305 cpu_manager.go:414] "containerMap: removing container" podUID="f19c63e0-0d0c-4b63-a59f-7aca9057efc2" containerName="revoke-keys" Nov 03 21:19:54.386476 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:54.386398 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f19c63e0-0d0c-4b63-a59f-7aca9057efc2" containerName="revoke-keys" Nov 03 21:19:54.386581 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:54.386498 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="f19c63e0-0d0c-4b63-a59f-7aca9057efc2" containerName="revoke-keys" Nov 03 21:19:54.389492 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:54.389477 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-czgbn" Nov 03 21:19:54.493798 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:54.493767 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:19:54.493996 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:54.493801 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77vtd\"" Nov 03 21:19:54.493996 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:54.493813 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:19:54.493996 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:54.493816 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:19:54.496000 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:54.495978 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-czgbn"] Nov 03 21:19:54.579106 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:54.579070 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/424f721f-ea02-4659-bfa6-7f11bb3a901d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-czgbn\" (UID: \"424f721f-ea02-4659-bfa6-7f11bb3a901d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-czgbn" Nov 03 21:19:54.579286 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:54.579158 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/424f721f-ea02-4659-bfa6-7f11bb3a901d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-czgbn\" (UID: \"424f721f-ea02-4659-bfa6-7f11bb3a901d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-czgbn" Nov 03 21:19:54.579337 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:54.579316 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms5m6\" (UniqueName: \"kubernetes.io/projected/424f721f-ea02-4659-bfa6-7f11bb3a901d-kube-api-access-ms5m6\") pod \"maas-api-revoke-sub-0b7e8edb-czgbn\" (UID: \"424f721f-ea02-4659-bfa6-7f11bb3a901d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-czgbn" Nov 03 21:19:54.680863 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:54.680824 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms5m6\" (UniqueName: \"kubernetes.io/projected/424f721f-ea02-4659-bfa6-7f11bb3a901d-kube-api-access-ms5m6\") pod \"maas-api-revoke-sub-0b7e8edb-czgbn\" (UID: \"424f721f-ea02-4659-bfa6-7f11bb3a901d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-czgbn" Nov 03 21:19:54.681070 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:54.680879 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/424f721f-ea02-4659-bfa6-7f11bb3a901d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-czgbn\" (UID: \"424f721f-ea02-4659-bfa6-7f11bb3a901d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-czgbn" Nov 03 21:19:54.681070 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:54.680924 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/424f721f-ea02-4659-bfa6-7f11bb3a901d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-czgbn\" (UID: \"424f721f-ea02-4659-bfa6-7f11bb3a901d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-czgbn" Nov 03 21:19:54.681588 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:54.681559 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/424f721f-ea02-4659-bfa6-7f11bb3a901d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-czgbn\" (UID: \"424f721f-ea02-4659-bfa6-7f11bb3a901d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-czgbn" Nov 03 21:19:54.683489 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:54.683469 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/424f721f-ea02-4659-bfa6-7f11bb3a901d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-czgbn\" (UID: \"424f721f-ea02-4659-bfa6-7f11bb3a901d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-czgbn" Nov 03 21:19:54.742245 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:54.742211 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms5m6\" (UniqueName: \"kubernetes.io/projected/424f721f-ea02-4659-bfa6-7f11bb3a901d-kube-api-access-ms5m6\") pod \"maas-api-revoke-sub-0b7e8edb-czgbn\" (UID: \"424f721f-ea02-4659-bfa6-7f11bb3a901d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-czgbn" Nov 03 21:19:55.002251 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:55.002170 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-czgbn" Nov 03 21:19:55.377459 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:55.377411 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-czgbn"] Nov 03 21:19:55.379180 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:19:55.379145 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod424f721f_ea02_4659_bfa6_7f11bb3a901d.slice/crio-8c0826d5756451ba5ecc3ce5cb3cb34ae360ba8993f8704731cffaa62b26f61f WatchSource:0}: Error finding container 8c0826d5756451ba5ecc3ce5cb3cb34ae360ba8993f8704731cffaa62b26f61f: Status 404 returned error can't find the container with id 8c0826d5756451ba5ecc3ce5cb3cb34ae360ba8993f8704731cffaa62b26f61f Nov 03 21:19:56.081146 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:56.081053 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-czgbn" event={"ID":"424f721f-ea02-4659-bfa6-7f11bb3a901d","Type":"ContainerStarted","Data":"36b2ee1052cffbc322ae23d2f50263c73afa39fb21ecdda7c9e92e7b28acda04"} Nov 03 21:19:56.081570 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:56.081152 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-czgbn" event={"ID":"424f721f-ea02-4659-bfa6-7f11bb3a901d","Type":"ContainerStarted","Data":"8c0826d5756451ba5ecc3ce5cb3cb34ae360ba8993f8704731cffaa62b26f61f"} Nov 03 21:19:56.197343 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:56.197284 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-czgbn" podStartSLOduration=2.197271372 podStartE2EDuration="2.197271372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:19:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:19:56.194934969 +0000 UTC m=+2431.420254777" watchObservedRunningTime="2026-11-03 21:19:56.197271372 +0000 UTC m=+2431.422591161" Nov 03 21:19:57.087920 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:57.087878 4305 generic.go:356] "Generic (PLEG): container finished" podID="424f721f-ea02-4659-bfa6-7f11bb3a901d" containerID="36b2ee1052cffbc322ae23d2f50263c73afa39fb21ecdda7c9e92e7b28acda04" exitCode=0 Nov 03 21:19:57.088357 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:57.087996 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-czgbn" event={"ID":"424f721f-ea02-4659-bfa6-7f11bb3a901d","Type":"ContainerDied","Data":"36b2ee1052cffbc322ae23d2f50263c73afa39fb21ecdda7c9e92e7b28acda04"} Nov 03 21:19:58.236105 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:58.236081 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-czgbn" Nov 03 21:19:58.322720 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:58.322690 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/424f721f-ea02-4659-bfa6-7f11bb3a901d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/424f721f-ea02-4659-bfa6-7f11bb3a901d-maas-api-openshift-ca\") pod \"424f721f-ea02-4659-bfa6-7f11bb3a901d\" (UID: \"424f721f-ea02-4659-bfa6-7f11bb3a901d\") " Nov 03 21:19:58.322900 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:58.322804 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/424f721f-ea02-4659-bfa6-7f11bb3a901d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/424f721f-ea02-4659-bfa6-7f11bb3a901d-maas-api-xks-ca\") pod \"424f721f-ea02-4659-bfa6-7f11bb3a901d\" (UID: \"424f721f-ea02-4659-bfa6-7f11bb3a901d\") " Nov 03 21:19:58.322900 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:58.322886 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/424f721f-ea02-4659-bfa6-7f11bb3a901d-kube-api-access-ms5m6\" (UniqueName: \"kubernetes.io/projected/424f721f-ea02-4659-bfa6-7f11bb3a901d-kube-api-access-ms5m6\") pod \"424f721f-ea02-4659-bfa6-7f11bb3a901d\" (UID: \"424f721f-ea02-4659-bfa6-7f11bb3a901d\") " Nov 03 21:19:58.323099 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:58.323078 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/424f721f-ea02-4659-bfa6-7f11bb3a901d-maas-api-openshift-ca" pod "424f721f-ea02-4659-bfa6-7f11bb3a901d" (UID: "424f721f-ea02-4659-bfa6-7f11bb3a901d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:19:58.323212 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:58.323198 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/424f721f-ea02-4659-bfa6-7f11bb3a901d-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:19:58.325051 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:58.325024 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/424f721f-ea02-4659-bfa6-7f11bb3a901d-kube-api-access-ms5m6" pod "424f721f-ea02-4659-bfa6-7f11bb3a901d" (UID: "424f721f-ea02-4659-bfa6-7f11bb3a901d"). InnerVolumeSpecName "kube-api-access-ms5m6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:19:58.325167 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:58.325107 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/424f721f-ea02-4659-bfa6-7f11bb3a901d-maas-api-xks-ca" pod "424f721f-ea02-4659-bfa6-7f11bb3a901d" (UID: "424f721f-ea02-4659-bfa6-7f11bb3a901d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:19:58.424113 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:58.424067 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/424f721f-ea02-4659-bfa6-7f11bb3a901d-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:19:58.424113 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:58.424106 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ms5m6\" (UniqueName: \"kubernetes.io/projected/424f721f-ea02-4659-bfa6-7f11bb3a901d-kube-api-access-ms5m6\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:19:59.100853 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:59.100821 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-czgbn" Nov 03 21:19:59.101040 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:59.100820 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-czgbn" event={"ID":"424f721f-ea02-4659-bfa6-7f11bb3a901d","Type":"ContainerDied","Data":"8c0826d5756451ba5ecc3ce5cb3cb34ae360ba8993f8704731cffaa62b26f61f"} Nov 03 21:19:59.101040 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:19:59.100947 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c0826d5756451ba5ecc3ce5cb3cb34ae360ba8993f8704731cffaa62b26f61f" Nov 03 21:20:05.757163 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:05.757120 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gvs5"] Nov 03 21:20:05.757810 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:05.757789 4305 cpu_manager.go:414] "containerMap: removing container" podUID="424f721f-ea02-4659-bfa6-7f11bb3a901d" containerName="revoke-keys" Nov 03 21:20:05.757872 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:05.757813 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="424f721f-ea02-4659-bfa6-7f11bb3a901d" containerName="revoke-keys" Nov 03 21:20:05.757938 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:05.757925 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="424f721f-ea02-4659-bfa6-7f11bb3a901d" containerName="revoke-keys" Nov 03 21:20:05.760974 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:05.760954 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gvs5" Nov 03 21:20:05.803189 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:05.803161 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvpf5\" (UniqueName: \"kubernetes.io/projected/a5536aa4-cf58-4054-a1c2-8514a158daea-kube-api-access-nvpf5\") pod \"maas-api-revoke-sub-3d4156f1-6gvs5\" (UID: \"a5536aa4-cf58-4054-a1c2-8514a158daea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gvs5" Nov 03 21:20:05.803389 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:05.803211 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5536aa4-cf58-4054-a1c2-8514a158daea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-6gvs5\" (UID: \"a5536aa4-cf58-4054-a1c2-8514a158daea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gvs5" Nov 03 21:20:05.803389 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:05.803360 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5536aa4-cf58-4054-a1c2-8514a158daea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-6gvs5\" (UID: \"a5536aa4-cf58-4054-a1c2-8514a158daea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gvs5" Nov 03 21:20:05.866818 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:05.866786 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:20:05.867013 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:05.866824 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77vtd\"" Nov 03 21:20:05.867013 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:05.866839 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:20:05.867013 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:05.866895 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:20:05.869029 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:05.869008 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gvs5"] Nov 03 21:20:05.904210 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:05.904166 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvpf5\" (UniqueName: \"kubernetes.io/projected/a5536aa4-cf58-4054-a1c2-8514a158daea-kube-api-access-nvpf5\") pod \"maas-api-revoke-sub-3d4156f1-6gvs5\" (UID: \"a5536aa4-cf58-4054-a1c2-8514a158daea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gvs5" Nov 03 21:20:05.904480 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:05.904234 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5536aa4-cf58-4054-a1c2-8514a158daea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-6gvs5\" (UID: \"a5536aa4-cf58-4054-a1c2-8514a158daea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gvs5" Nov 03 21:20:05.904480 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:05.904269 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5536aa4-cf58-4054-a1c2-8514a158daea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-6gvs5\" (UID: \"a5536aa4-cf58-4054-a1c2-8514a158daea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gvs5" Nov 03 21:20:05.905053 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:05.905026 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5536aa4-cf58-4054-a1c2-8514a158daea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-6gvs5\" (UID: \"a5536aa4-cf58-4054-a1c2-8514a158daea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gvs5" Nov 03 21:20:05.906747 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:05.906728 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5536aa4-cf58-4054-a1c2-8514a158daea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-6gvs5\" (UID: \"a5536aa4-cf58-4054-a1c2-8514a158daea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gvs5" Nov 03 21:20:05.965374 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:05.965337 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvpf5\" (UniqueName: \"kubernetes.io/projected/a5536aa4-cf58-4054-a1c2-8514a158daea-kube-api-access-nvpf5\") pod \"maas-api-revoke-sub-3d4156f1-6gvs5\" (UID: \"a5536aa4-cf58-4054-a1c2-8514a158daea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gvs5" Nov 03 21:20:06.075838 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:06.075740 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gvs5" Nov 03 21:20:06.459101 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:06.459046 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gvs5"] Nov 03 21:20:06.461602 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:20:06.461568 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5536aa4_cf58_4054_a1c2_8514a158daea.slice/crio-5c5ca331b89c858e8a54bf726945d683583fd6dc18a760a21e5ba3c146b2af47 WatchSource:0}: Error finding container 5c5ca331b89c858e8a54bf726945d683583fd6dc18a760a21e5ba3c146b2af47: Status 404 returned error can't find the container with id 5c5ca331b89c858e8a54bf726945d683583fd6dc18a760a21e5ba3c146b2af47 Nov 03 21:20:07.161484 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:07.161417 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gvs5" event={"ID":"a5536aa4-cf58-4054-a1c2-8514a158daea","Type":"ContainerStarted","Data":"6e59c7d74890f7a5e32add8b385488d0996c40a2b28d28519832ddeeaaed2bc8"} Nov 03 21:20:07.161903 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:07.161507 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gvs5" event={"ID":"a5536aa4-cf58-4054-a1c2-8514a158daea","Type":"ContainerStarted","Data":"5c5ca331b89c858e8a54bf726945d683583fd6dc18a760a21e5ba3c146b2af47"} Nov 03 21:20:07.278472 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:07.278406 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gvs5" podStartSLOduration=2.2783896759999998 podStartE2EDuration="2.278389676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:20:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:20:07.275467409 +0000 UTC m=+2442.500787215" watchObservedRunningTime="2026-11-03 21:20:07.278389676 +0000 UTC m=+2442.503709465" Nov 03 21:20:07.327871 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:07.327836 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:20:08.168408 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:08.168372 4305 generic.go:356] "Generic (PLEG): container finished" podID="a5536aa4-cf58-4054-a1c2-8514a158daea" containerID="6e59c7d74890f7a5e32add8b385488d0996c40a2b28d28519832ddeeaaed2bc8" exitCode=0 Nov 03 21:20:08.168861 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:08.168465 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gvs5" event={"ID":"a5536aa4-cf58-4054-a1c2-8514a158daea","Type":"ContainerDied","Data":"6e59c7d74890f7a5e32add8b385488d0996c40a2b28d28519832ddeeaaed2bc8"} Nov 03 21:20:09.318496 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:09.318465 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gvs5" Nov 03 21:20:09.442833 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:09.442750 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5536aa4-cf58-4054-a1c2-8514a158daea-kube-api-access-nvpf5\" (UniqueName: \"kubernetes.io/projected/a5536aa4-cf58-4054-a1c2-8514a158daea-kube-api-access-nvpf5\") pod \"a5536aa4-cf58-4054-a1c2-8514a158daea\" (UID: \"a5536aa4-cf58-4054-a1c2-8514a158daea\") " Nov 03 21:20:09.443020 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:09.442854 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5536aa4-cf58-4054-a1c2-8514a158daea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5536aa4-cf58-4054-a1c2-8514a158daea-maas-api-xks-ca\") pod \"a5536aa4-cf58-4054-a1c2-8514a158daea\" (UID: \"a5536aa4-cf58-4054-a1c2-8514a158daea\") " Nov 03 21:20:09.443020 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:09.442912 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5536aa4-cf58-4054-a1c2-8514a158daea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5536aa4-cf58-4054-a1c2-8514a158daea-maas-api-openshift-ca\") pod \"a5536aa4-cf58-4054-a1c2-8514a158daea\" (UID: \"a5536aa4-cf58-4054-a1c2-8514a158daea\") " Nov 03 21:20:09.443621 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:09.443597 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5536aa4-cf58-4054-a1c2-8514a158daea-maas-api-openshift-ca" pod "a5536aa4-cf58-4054-a1c2-8514a158daea" (UID: "a5536aa4-cf58-4054-a1c2-8514a158daea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:20:09.445142 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:09.445117 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5536aa4-cf58-4054-a1c2-8514a158daea-kube-api-access-nvpf5" pod "a5536aa4-cf58-4054-a1c2-8514a158daea" (UID: "a5536aa4-cf58-4054-a1c2-8514a158daea"). InnerVolumeSpecName "kube-api-access-nvpf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:20:09.445320 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:09.445298 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5536aa4-cf58-4054-a1c2-8514a158daea-maas-api-xks-ca" pod "a5536aa4-cf58-4054-a1c2-8514a158daea" (UID: "a5536aa4-cf58-4054-a1c2-8514a158daea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:20:09.544568 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:09.544534 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvpf5\" (UniqueName: \"kubernetes.io/projected/a5536aa4-cf58-4054-a1c2-8514a158daea-kube-api-access-nvpf5\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:20:09.544568 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:09.544564 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5536aa4-cf58-4054-a1c2-8514a158daea-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:20:09.544568 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:09.544574 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5536aa4-cf58-4054-a1c2-8514a158daea-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:20:10.181063 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:10.181036 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gvs5" Nov 03 21:20:10.181260 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:10.181032 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gvs5" event={"ID":"a5536aa4-cf58-4054-a1c2-8514a158daea","Type":"ContainerDied","Data":"5c5ca331b89c858e8a54bf726945d683583fd6dc18a760a21e5ba3c146b2af47"} Nov 03 21:20:10.181260 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:10.181170 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c5ca331b89c858e8a54bf726945d683583fd6dc18a760a21e5ba3c146b2af47" Nov 03 21:20:15.048169 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:15.048130 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-trmz9"] Nov 03 21:20:15.097404 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:15.097368 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-trmz9"] Nov 03 21:20:16.320585 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:20:16.320535 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:20:16.895087 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:16.895047 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44bb69ae-9081-4986-a048-5fe5c3c89ff4" path="/var/lib/kubelet/pods/44bb69ae-9081-4986-a048-5fe5c3c89ff4/volumes" Nov 03 21:20:20.403445 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:20.403391 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:20:31.178341 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:31.178305 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:20:32.723813 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:32.723780 4305 scope.go:122] "RemoveContainer" containerID="d7f0175080aa64f5f922196e3f06e75d7c3d0cefc1992fc3fdc1859cf2744cf1" Nov 03 21:20:32.732631 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:32.732610 4305 scope.go:122] "RemoveContainer" containerID="9fa7902258a909feffc1ebd2b8c620199f3b6eb406d744db68313d4ec51cf02c" Nov 03 21:20:32.740769 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:32.740751 4305 scope.go:122] "RemoveContainer" containerID="bee3d344b6ddf6af03c750bb33e3fdb33209b01afbe44a1a21780d2bcd4ba8e3" Nov 03 21:20:32.748703 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:32.748677 4305 scope.go:122] "RemoveContainer" containerID="4b6e94af048ab5358be8f1942e21d34c8104c475f0246e1367864ab94a7cc6e1" Nov 03 21:20:32.756426 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:32.756407 4305 scope.go:122] "RemoveContainer" containerID="ce3da5d587a8ad4a8e661b41a01eb92efbec6492373146c315a943890f8bc78d" Nov 03 21:20:32.764419 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:32.764396 4305 scope.go:122] "RemoveContainer" containerID="658b0fafc8b622185d9dd861446d4b1e2b159b351dc530867ddd934c3daf88be" Nov 03 21:20:32.772215 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:32.772193 4305 scope.go:122] "RemoveContainer" containerID="bb2a68447ca969d2a8aa6fd28ff9cd774d3af40c41d10c351a099fb5d20fe7bc" Nov 03 21:20:32.779928 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:32.779900 4305 scope.go:122] "RemoveContainer" containerID="59e7ac1ddd423b18db71868de5bc2d66bf460e7e17d060598d6b128b15e8d0d0" Nov 03 21:20:32.787894 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:32.787874 4305 scope.go:122] "RemoveContainer" containerID="e8ae5aed3065eceee49db48d8fe2bb21e9fc417af7c418d8912df1c7d3d00c4a" Nov 03 21:20:39.384589 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:39.384510 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gvs5"] Nov 03 21:20:39.434234 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:39.434202 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6gvs5"] Nov 03 21:20:39.508300 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:39.508270 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zsds5"] Nov 03 21:20:39.508732 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:39.508718 4305 cpu_manager.go:414] "containerMap: removing container" podUID="a5536aa4-cf58-4054-a1c2-8514a158daea" containerName="revoke-keys" Nov 03 21:20:39.508782 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:39.508733 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5536aa4-cf58-4054-a1c2-8514a158daea" containerName="revoke-keys" Nov 03 21:20:39.508837 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:39.508829 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5536aa4-cf58-4054-a1c2-8514a158daea" containerName="revoke-keys" Nov 03 21:20:39.513997 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:39.513979 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zsds5" Nov 03 21:20:39.555052 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:39.555022 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/955a7325-3635-42b0-9bc3-87a06f1a6763-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-zsds5\" (UID: \"955a7325-3635-42b0-9bc3-87a06f1a6763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zsds5" Nov 03 21:20:39.555233 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:39.555154 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cxfx\" (UniqueName: \"kubernetes.io/projected/955a7325-3635-42b0-9bc3-87a06f1a6763-kube-api-access-5cxfx\") pod \"maas-api-revoke-sub-3d4156f1-zsds5\" (UID: \"955a7325-3635-42b0-9bc3-87a06f1a6763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zsds5" Nov 03 21:20:39.555233 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:39.555194 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/955a7325-3635-42b0-9bc3-87a06f1a6763-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-zsds5\" (UID: \"955a7325-3635-42b0-9bc3-87a06f1a6763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zsds5" Nov 03 21:20:39.566491 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:39.566410 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:20:39.566491 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:39.566411 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:20:39.566491 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:39.566491 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:20:39.566791 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:39.566621 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77vtd\"" Nov 03 21:20:39.569655 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:39.569630 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zsds5"] Nov 03 21:20:39.656757 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:39.656722 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cxfx\" (UniqueName: \"kubernetes.io/projected/955a7325-3635-42b0-9bc3-87a06f1a6763-kube-api-access-5cxfx\") pod \"maas-api-revoke-sub-3d4156f1-zsds5\" (UID: \"955a7325-3635-42b0-9bc3-87a06f1a6763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zsds5" Nov 03 21:20:39.656757 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:39.656760 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/955a7325-3635-42b0-9bc3-87a06f1a6763-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-zsds5\" (UID: \"955a7325-3635-42b0-9bc3-87a06f1a6763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zsds5" Nov 03 21:20:39.656997 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:39.656840 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/955a7325-3635-42b0-9bc3-87a06f1a6763-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-zsds5\" (UID: \"955a7325-3635-42b0-9bc3-87a06f1a6763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zsds5" Nov 03 21:20:39.657497 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:39.657477 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/955a7325-3635-42b0-9bc3-87a06f1a6763-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-zsds5\" (UID: \"955a7325-3635-42b0-9bc3-87a06f1a6763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zsds5" Nov 03 21:20:39.659321 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:39.659298 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/955a7325-3635-42b0-9bc3-87a06f1a6763-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-zsds5\" (UID: \"955a7325-3635-42b0-9bc3-87a06f1a6763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zsds5" Nov 03 21:20:39.718610 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:39.718577 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cxfx\" (UniqueName: \"kubernetes.io/projected/955a7325-3635-42b0-9bc3-87a06f1a6763-kube-api-access-5cxfx\") pod \"maas-api-revoke-sub-3d4156f1-zsds5\" (UID: \"955a7325-3635-42b0-9bc3-87a06f1a6763\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zsds5" Nov 03 21:20:39.827086 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:39.827052 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zsds5" Nov 03 21:20:40.205227 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:40.205191 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zsds5"] Nov 03 21:20:40.208781 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:20:40.208751 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod955a7325_3635_42b0_9bc3_87a06f1a6763.slice/crio-7080ac4f871ae8cc1cfe9e418c2da3d6fd81b350753c03bc9e3abc9e6559d3f8 WatchSource:0}: Error finding container 7080ac4f871ae8cc1cfe9e418c2da3d6fd81b350753c03bc9e3abc9e6559d3f8: Status 404 returned error can't find the container with id 7080ac4f871ae8cc1cfe9e418c2da3d6fd81b350753c03bc9e3abc9e6559d3f8 Nov 03 21:20:40.371692 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:40.371658 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zsds5" event={"ID":"955a7325-3635-42b0-9bc3-87a06f1a6763","Type":"ContainerStarted","Data":"7080ac4f871ae8cc1cfe9e418c2da3d6fd81b350753c03bc9e3abc9e6559d3f8"} Nov 03 21:20:40.894101 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:40.894075 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5536aa4-cf58-4054-a1c2-8514a158daea" path="/var/lib/kubelet/pods/a5536aa4-cf58-4054-a1c2-8514a158daea/volumes" Nov 03 21:20:41.379078 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:41.378978 4305 generic.go:356] "Generic (PLEG): container finished" podID="955a7325-3635-42b0-9bc3-87a06f1a6763" containerID="f728d4976ad4d0cd1070055db8ba188918e84c8f691a74cf33990f9d9070b372" exitCode=0 Nov 03 21:20:41.379078 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:41.379042 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zsds5" event={"ID":"955a7325-3635-42b0-9bc3-87a06f1a6763","Type":"ContainerDied","Data":"f728d4976ad4d0cd1070055db8ba188918e84c8f691a74cf33990f9d9070b372"} Nov 03 21:20:42.521871 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:42.521845 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zsds5" Nov 03 21:20:42.585793 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:42.585763 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/955a7325-3635-42b0-9bc3-87a06f1a6763-kube-api-access-5cxfx\" (UniqueName: \"kubernetes.io/projected/955a7325-3635-42b0-9bc3-87a06f1a6763-kube-api-access-5cxfx\") pod \"955a7325-3635-42b0-9bc3-87a06f1a6763\" (UID: \"955a7325-3635-42b0-9bc3-87a06f1a6763\") " Nov 03 21:20:42.585964 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:42.585880 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/955a7325-3635-42b0-9bc3-87a06f1a6763-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/955a7325-3635-42b0-9bc3-87a06f1a6763-maas-api-openshift-ca\") pod \"955a7325-3635-42b0-9bc3-87a06f1a6763\" (UID: \"955a7325-3635-42b0-9bc3-87a06f1a6763\") " Nov 03 21:20:42.585964 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:42.585914 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/955a7325-3635-42b0-9bc3-87a06f1a6763-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/955a7325-3635-42b0-9bc3-87a06f1a6763-maas-api-xks-ca\") pod \"955a7325-3635-42b0-9bc3-87a06f1a6763\" (UID: \"955a7325-3635-42b0-9bc3-87a06f1a6763\") " Nov 03 21:20:42.586236 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:42.586210 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/955a7325-3635-42b0-9bc3-87a06f1a6763-maas-api-openshift-ca" pod "955a7325-3635-42b0-9bc3-87a06f1a6763" (UID: "955a7325-3635-42b0-9bc3-87a06f1a6763"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:20:42.588007 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:42.587976 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/955a7325-3635-42b0-9bc3-87a06f1a6763-kube-api-access-5cxfx" pod "955a7325-3635-42b0-9bc3-87a06f1a6763" (UID: "955a7325-3635-42b0-9bc3-87a06f1a6763"). InnerVolumeSpecName "kube-api-access-5cxfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:20:42.588102 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:42.588054 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/955a7325-3635-42b0-9bc3-87a06f1a6763-maas-api-xks-ca" pod "955a7325-3635-42b0-9bc3-87a06f1a6763" (UID: "955a7325-3635-42b0-9bc3-87a06f1a6763"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:20:42.687357 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:42.687315 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cxfx\" (UniqueName: \"kubernetes.io/projected/955a7325-3635-42b0-9bc3-87a06f1a6763-kube-api-access-5cxfx\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:20:42.687357 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:42.687349 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/955a7325-3635-42b0-9bc3-87a06f1a6763-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:20:42.687357 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:42.687363 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/955a7325-3635-42b0-9bc3-87a06f1a6763-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:20:43.390370 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:43.390340 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zsds5" Nov 03 21:20:43.390370 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:43.390359 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zsds5" event={"ID":"955a7325-3635-42b0-9bc3-87a06f1a6763","Type":"ContainerDied","Data":"7080ac4f871ae8cc1cfe9e418c2da3d6fd81b350753c03bc9e3abc9e6559d3f8"} Nov 03 21:20:43.390626 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:43.390406 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7080ac4f871ae8cc1cfe9e418c2da3d6fd81b350753c03bc9e3abc9e6559d3f8" Nov 03 21:20:46.321203 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:20:46.321169 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:20:52.388633 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:20:52.388594 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:21:03.514400 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:03.514362 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:21:16.321044 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:21:16.321005 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:21:20.280812 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:20.280777 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:21:30.187894 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:30.187857 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qthgc"] Nov 03 21:21:30.188418 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:30.188399 4305 cpu_manager.go:414] "containerMap: removing container" podUID="955a7325-3635-42b0-9bc3-87a06f1a6763" containerName="revoke-keys" Nov 03 21:21:30.188510 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:30.188421 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="955a7325-3635-42b0-9bc3-87a06f1a6763" containerName="revoke-keys" Nov 03 21:21:30.188561 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:30.188527 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="955a7325-3635-42b0-9bc3-87a06f1a6763" containerName="revoke-keys" Nov 03 21:21:30.191849 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:30.191834 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qthgc" Nov 03 21:21:30.210056 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:30.210021 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/611e686a-7db8-4a27-9798-e098067154fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-qthgc\" (UID: \"611e686a-7db8-4a27-9798-e098067154fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qthgc" Nov 03 21:21:30.210216 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:30.210128 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9c9c\" (UniqueName: \"kubernetes.io/projected/611e686a-7db8-4a27-9798-e098067154fe-kube-api-access-d9c9c\") pod \"maas-api-revoke-sub-e2fc9c5b-qthgc\" (UID: \"611e686a-7db8-4a27-9798-e098067154fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qthgc" Nov 03 21:21:30.210216 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:30.210178 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/611e686a-7db8-4a27-9798-e098067154fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-qthgc\" (UID: \"611e686a-7db8-4a27-9798-e098067154fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qthgc" Nov 03 21:21:30.249024 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:30.248983 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qthgc"] Nov 03 21:21:30.295025 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:30.294987 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:21:30.295025 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:30.295000 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:21:30.295260 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:30.295031 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77vtd\"" Nov 03 21:21:30.295260 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:30.295060 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:21:30.311607 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:30.311575 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/611e686a-7db8-4a27-9798-e098067154fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-qthgc\" (UID: \"611e686a-7db8-4a27-9798-e098067154fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qthgc" Nov 03 21:21:30.311781 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:30.311658 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/611e686a-7db8-4a27-9798-e098067154fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-qthgc\" (UID: \"611e686a-7db8-4a27-9798-e098067154fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qthgc" Nov 03 21:21:30.311781 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:30.311707 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9c9c\" (UniqueName: \"kubernetes.io/projected/611e686a-7db8-4a27-9798-e098067154fe-kube-api-access-d9c9c\") pod \"maas-api-revoke-sub-e2fc9c5b-qthgc\" (UID: \"611e686a-7db8-4a27-9798-e098067154fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qthgc" Nov 03 21:21:30.312310 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:30.312285 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/611e686a-7db8-4a27-9798-e098067154fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-qthgc\" (UID: \"611e686a-7db8-4a27-9798-e098067154fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qthgc" Nov 03 21:21:30.314185 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:30.314166 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/611e686a-7db8-4a27-9798-e098067154fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-qthgc\" (UID: \"611e686a-7db8-4a27-9798-e098067154fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qthgc" Nov 03 21:21:30.373276 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:30.373229 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9c9c\" (UniqueName: \"kubernetes.io/projected/611e686a-7db8-4a27-9798-e098067154fe-kube-api-access-d9c9c\") pod \"maas-api-revoke-sub-e2fc9c5b-qthgc\" (UID: \"611e686a-7db8-4a27-9798-e098067154fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qthgc" Nov 03 21:21:30.505697 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:30.505596 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qthgc" Nov 03 21:21:30.879017 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:30.878993 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qthgc"] Nov 03 21:21:30.881271 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:21:30.881240 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod611e686a_7db8_4a27_9798_e098067154fe.slice/crio-57f228c6be139f097083384690aaa3721214d3f0721b900618d20decd0460135 WatchSource:0}: Error finding container 57f228c6be139f097083384690aaa3721214d3f0721b900618d20decd0460135: Status 404 returned error can't find the container with id 57f228c6be139f097083384690aaa3721214d3f0721b900618d20decd0460135 Nov 03 21:21:31.674729 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:31.674694 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qthgc" event={"ID":"611e686a-7db8-4a27-9798-e098067154fe","Type":"ContainerStarted","Data":"05265995defacb071668c0ef2e7821ccaad8b0c23049ee7f7dffa3f255926944"} Nov 03 21:21:31.675155 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:31.674754 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qthgc" event={"ID":"611e686a-7db8-4a27-9798-e098067154fe","Type":"ContainerStarted","Data":"57f228c6be139f097083384690aaa3721214d3f0721b900618d20decd0460135"} Nov 03 21:21:31.794533 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:31.794475 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qthgc" podStartSLOduration=1.7944630190000002 podStartE2EDuration="1.794463019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:21:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:21:31.793956441 +0000 UTC m=+2527.019276231" watchObservedRunningTime="2026-11-03 21:21:31.794463019 +0000 UTC m=+2527.019782808" Nov 03 21:21:32.681477 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:32.681417 4305 generic.go:356] "Generic (PLEG): container finished" podID="611e686a-7db8-4a27-9798-e098067154fe" containerID="05265995defacb071668c0ef2e7821ccaad8b0c23049ee7f7dffa3f255926944" exitCode=0 Nov 03 21:21:32.681900 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:32.681516 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qthgc" event={"ID":"611e686a-7db8-4a27-9798-e098067154fe","Type":"ContainerDied","Data":"05265995defacb071668c0ef2e7821ccaad8b0c23049ee7f7dffa3f255926944"} Nov 03 21:21:32.839061 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:32.839035 4305 scope.go:122] "RemoveContainer" containerID="d77c119af204499e3ffd6914d0304c5e7b0b65ef421ece23d6d32a2a05a9b362" Nov 03 21:21:33.843605 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:33.843577 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qthgc" Nov 03 21:21:33.948119 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:33.948031 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/611e686a-7db8-4a27-9798-e098067154fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/611e686a-7db8-4a27-9798-e098067154fe-maas-api-xks-ca\") pod \"611e686a-7db8-4a27-9798-e098067154fe\" (UID: \"611e686a-7db8-4a27-9798-e098067154fe\") " Nov 03 21:21:33.948119 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:33.948089 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/611e686a-7db8-4a27-9798-e098067154fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/611e686a-7db8-4a27-9798-e098067154fe-maas-api-openshift-ca\") pod \"611e686a-7db8-4a27-9798-e098067154fe\" (UID: \"611e686a-7db8-4a27-9798-e098067154fe\") " Nov 03 21:21:33.948327 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:33.948146 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/611e686a-7db8-4a27-9798-e098067154fe-kube-api-access-d9c9c\" (UniqueName: \"kubernetes.io/projected/611e686a-7db8-4a27-9798-e098067154fe-kube-api-access-d9c9c\") pod \"611e686a-7db8-4a27-9798-e098067154fe\" (UID: \"611e686a-7db8-4a27-9798-e098067154fe\") " Nov 03 21:21:33.948492 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:33.948461 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/611e686a-7db8-4a27-9798-e098067154fe-maas-api-openshift-ca" pod "611e686a-7db8-4a27-9798-e098067154fe" (UID: "611e686a-7db8-4a27-9798-e098067154fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:21:33.948712 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:33.948696 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/611e686a-7db8-4a27-9798-e098067154fe-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:21:33.950408 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:33.950387 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/611e686a-7db8-4a27-9798-e098067154fe-maas-api-xks-ca" pod "611e686a-7db8-4a27-9798-e098067154fe" (UID: "611e686a-7db8-4a27-9798-e098067154fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:21:33.950519 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:33.950472 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/611e686a-7db8-4a27-9798-e098067154fe-kube-api-access-d9c9c" pod "611e686a-7db8-4a27-9798-e098067154fe" (UID: "611e686a-7db8-4a27-9798-e098067154fe"). InnerVolumeSpecName "kube-api-access-d9c9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:21:34.049765 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:34.049723 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9c9c\" (UniqueName: \"kubernetes.io/projected/611e686a-7db8-4a27-9798-e098067154fe-kube-api-access-d9c9c\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:21:34.049765 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:34.049757 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/611e686a-7db8-4a27-9798-e098067154fe-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:21:34.693125 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:34.693090 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qthgc" event={"ID":"611e686a-7db8-4a27-9798-e098067154fe","Type":"ContainerDied","Data":"57f228c6be139f097083384690aaa3721214d3f0721b900618d20decd0460135"} Nov 03 21:21:34.693125 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:34.693121 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qthgc" Nov 03 21:21:34.693356 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:34.693142 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57f228c6be139f097083384690aaa3721214d3f0721b900618d20decd0460135" Nov 03 21:21:40.942535 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:40.942492 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-l7t5m"] Nov 03 21:21:40.942947 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:40.942928 4305 cpu_manager.go:414] "containerMap: removing container" podUID="611e686a-7db8-4a27-9798-e098067154fe" containerName="revoke-keys" Nov 03 21:21:40.942947 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:40.942938 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="611e686a-7db8-4a27-9798-e098067154fe" containerName="revoke-keys" Nov 03 21:21:40.943040 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:40.943004 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="611e686a-7db8-4a27-9798-e098067154fe" containerName="revoke-keys" Nov 03 21:21:40.946283 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:40.946267 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-l7t5m" Nov 03 21:21:41.027346 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:41.027307 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/779c1871-8b66-477c-b9b4-1bb706571d65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-l7t5m\" (UID: \"779c1871-8b66-477c-b9b4-1bb706571d65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-l7t5m" Nov 03 21:21:41.027576 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:41.027492 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnhds\" (UniqueName: \"kubernetes.io/projected/779c1871-8b66-477c-b9b4-1bb706571d65-kube-api-access-jnhds\") pod \"maas-api-revoke-sub-5a29efb3-l7t5m\" (UID: \"779c1871-8b66-477c-b9b4-1bb706571d65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-l7t5m" Nov 03 21:21:41.027576 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:41.027557 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/779c1871-8b66-477c-b9b4-1bb706571d65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-l7t5m\" (UID: \"779c1871-8b66-477c-b9b4-1bb706571d65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-l7t5m" Nov 03 21:21:41.051062 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:41.051031 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:21:41.051264 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:41.051155 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:21:41.051264 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:41.051161 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77vtd\"" Nov 03 21:21:41.051264 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:41.051176 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:21:41.053569 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:41.053549 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-l7t5m"] Nov 03 21:21:41.129039 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:41.129002 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnhds\" (UniqueName: \"kubernetes.io/projected/779c1871-8b66-477c-b9b4-1bb706571d65-kube-api-access-jnhds\") pod \"maas-api-revoke-sub-5a29efb3-l7t5m\" (UID: \"779c1871-8b66-477c-b9b4-1bb706571d65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-l7t5m" Nov 03 21:21:41.129270 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:41.129075 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/779c1871-8b66-477c-b9b4-1bb706571d65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-l7t5m\" (UID: \"779c1871-8b66-477c-b9b4-1bb706571d65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-l7t5m" Nov 03 21:21:41.129270 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:41.129170 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/779c1871-8b66-477c-b9b4-1bb706571d65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-l7t5m\" (UID: \"779c1871-8b66-477c-b9b4-1bb706571d65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-l7t5m" Nov 03 21:21:41.129858 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:41.129827 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/779c1871-8b66-477c-b9b4-1bb706571d65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-l7t5m\" (UID: \"779c1871-8b66-477c-b9b4-1bb706571d65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-l7t5m" Nov 03 21:21:41.131722 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:41.131705 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/779c1871-8b66-477c-b9b4-1bb706571d65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-l7t5m\" (UID: \"779c1871-8b66-477c-b9b4-1bb706571d65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-l7t5m" Nov 03 21:21:41.190231 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:41.190204 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnhds\" (UniqueName: \"kubernetes.io/projected/779c1871-8b66-477c-b9b4-1bb706571d65-kube-api-access-jnhds\") pod \"maas-api-revoke-sub-5a29efb3-l7t5m\" (UID: \"779c1871-8b66-477c-b9b4-1bb706571d65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-l7t5m" Nov 03 21:21:41.259823 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:41.259729 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-l7t5m" Nov 03 21:21:41.632470 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:41.632423 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-l7t5m"] Nov 03 21:21:41.634740 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:21:41.634711 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod779c1871_8b66_477c_b9b4_1bb706571d65.slice/crio-76ab1200ad343e3f6ab5d6d7f712ae371f53279d0ab8dea22e2a5be8150263c3 WatchSource:0}: Error finding container 76ab1200ad343e3f6ab5d6d7f712ae371f53279d0ab8dea22e2a5be8150263c3: Status 404 returned error can't find the container with id 76ab1200ad343e3f6ab5d6d7f712ae371f53279d0ab8dea22e2a5be8150263c3 Nov 03 21:21:41.729967 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:41.729936 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-l7t5m" event={"ID":"779c1871-8b66-477c-b9b4-1bb706571d65","Type":"ContainerStarted","Data":"76ab1200ad343e3f6ab5d6d7f712ae371f53279d0ab8dea22e2a5be8150263c3"} Nov 03 21:21:42.736496 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:42.736456 4305 generic.go:356] "Generic (PLEG): container finished" podID="779c1871-8b66-477c-b9b4-1bb706571d65" containerID="b6c0b523326e29bf721f03529223ae454b06ecea7a9d6d59086ddf8969807b44" exitCode=0 Nov 03 21:21:42.736927 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:42.736537 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-l7t5m" event={"ID":"779c1871-8b66-477c-b9b4-1bb706571d65","Type":"ContainerDied","Data":"b6c0b523326e29bf721f03529223ae454b06ecea7a9d6d59086ddf8969807b44"} Nov 03 21:21:43.083485 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:43.083446 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:21:43.908312 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:43.908279 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-l7t5m" Nov 03 21:21:44.061662 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:44.061554 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/779c1871-8b66-477c-b9b4-1bb706571d65-kube-api-access-jnhds\" (UniqueName: \"kubernetes.io/projected/779c1871-8b66-477c-b9b4-1bb706571d65-kube-api-access-jnhds\") pod \"779c1871-8b66-477c-b9b4-1bb706571d65\" (UID: \"779c1871-8b66-477c-b9b4-1bb706571d65\") " Nov 03 21:21:44.061837 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:44.061684 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/779c1871-8b66-477c-b9b4-1bb706571d65-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/779c1871-8b66-477c-b9b4-1bb706571d65-maas-api-openshift-ca\") pod \"779c1871-8b66-477c-b9b4-1bb706571d65\" (UID: \"779c1871-8b66-477c-b9b4-1bb706571d65\") " Nov 03 21:21:44.061837 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:44.061736 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/779c1871-8b66-477c-b9b4-1bb706571d65-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/779c1871-8b66-477c-b9b4-1bb706571d65-maas-api-xks-ca\") pod \"779c1871-8b66-477c-b9b4-1bb706571d65\" (UID: \"779c1871-8b66-477c-b9b4-1bb706571d65\") " Nov 03 21:21:44.062389 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:44.062289 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/779c1871-8b66-477c-b9b4-1bb706571d65-maas-api-openshift-ca" pod "779c1871-8b66-477c-b9b4-1bb706571d65" (UID: "779c1871-8b66-477c-b9b4-1bb706571d65"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:21:44.064801 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:44.064760 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/779c1871-8b66-477c-b9b4-1bb706571d65-maas-api-xks-ca" pod "779c1871-8b66-477c-b9b4-1bb706571d65" (UID: "779c1871-8b66-477c-b9b4-1bb706571d65"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:21:44.065066 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:44.065032 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/779c1871-8b66-477c-b9b4-1bb706571d65-kube-api-access-jnhds" pod "779c1871-8b66-477c-b9b4-1bb706571d65" (UID: "779c1871-8b66-477c-b9b4-1bb706571d65"). InnerVolumeSpecName "kube-api-access-jnhds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:21:44.163856 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:44.163807 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnhds\" (UniqueName: \"kubernetes.io/projected/779c1871-8b66-477c-b9b4-1bb706571d65-kube-api-access-jnhds\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:21:44.163856 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:44.163855 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/779c1871-8b66-477c-b9b4-1bb706571d65-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:21:44.164170 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:44.163871 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/779c1871-8b66-477c-b9b4-1bb706571d65-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:21:44.750158 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:44.750131 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-l7t5m" Nov 03 21:21:44.750324 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:44.750131 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-l7t5m" event={"ID":"779c1871-8b66-477c-b9b4-1bb706571d65","Type":"ContainerDied","Data":"76ab1200ad343e3f6ab5d6d7f712ae371f53279d0ab8dea22e2a5be8150263c3"} Nov 03 21:21:44.750324 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:44.750262 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76ab1200ad343e3f6ab5d6d7f712ae371f53279d0ab8dea22e2a5be8150263c3" Nov 03 21:21:46.321011 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:21:46.320975 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:21:53.708526 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:53.708490 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:21:58.755210 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:58.755174 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zsds5"] Nov 03 21:21:58.804383 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:58.804341 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zsds5"] Nov 03 21:21:58.872337 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:58.872299 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwvw"] Nov 03 21:21:58.872746 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:58.872732 4305 cpu_manager.go:414] "containerMap: removing container" podUID="779c1871-8b66-477c-b9b4-1bb706571d65" containerName="revoke-keys" Nov 03 21:21:58.872804 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:58.872747 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="779c1871-8b66-477c-b9b4-1bb706571d65" containerName="revoke-keys" Nov 03 21:21:58.872843 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:58.872836 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="779c1871-8b66-477c-b9b4-1bb706571d65" containerName="revoke-keys" Nov 03 21:21:58.876160 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:58.876140 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwvw" Nov 03 21:21:58.895272 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:58.895244 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="955a7325-3635-42b0-9bc3-87a06f1a6763" path="/var/lib/kubelet/pods/955a7325-3635-42b0-9bc3-87a06f1a6763/volumes" Nov 03 21:21:58.930773 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:58.930732 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:21:58.931023 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:58.930831 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:21:58.931126 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:58.930928 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77vtd\"" Nov 03 21:21:58.931126 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:58.931022 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:21:58.933704 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:58.933674 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwvw"] Nov 03 21:21:58.961849 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:58.961815 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4vlc\" (UniqueName: \"kubernetes.io/projected/ee9107ea-d216-43a0-806c-2fc39c280e69-kube-api-access-g4vlc\") pod \"maas-api-revoke-sub-3d4156f1-ckwvw\" (UID: \"ee9107ea-d216-43a0-806c-2fc39c280e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwvw" Nov 03 21:21:58.962065 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:58.962025 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee9107ea-d216-43a0-806c-2fc39c280e69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ckwvw\" (UID: \"ee9107ea-d216-43a0-806c-2fc39c280e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwvw" Nov 03 21:21:58.962179 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:58.962160 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee9107ea-d216-43a0-806c-2fc39c280e69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ckwvw\" (UID: \"ee9107ea-d216-43a0-806c-2fc39c280e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwvw" Nov 03 21:21:59.063654 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:59.063562 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee9107ea-d216-43a0-806c-2fc39c280e69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ckwvw\" (UID: \"ee9107ea-d216-43a0-806c-2fc39c280e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwvw" Nov 03 21:21:59.063654 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:59.063616 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee9107ea-d216-43a0-806c-2fc39c280e69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ckwvw\" (UID: \"ee9107ea-d216-43a0-806c-2fc39c280e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwvw" Nov 03 21:21:59.063885 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:59.063680 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4vlc\" (UniqueName: \"kubernetes.io/projected/ee9107ea-d216-43a0-806c-2fc39c280e69-kube-api-access-g4vlc\") pod \"maas-api-revoke-sub-3d4156f1-ckwvw\" (UID: \"ee9107ea-d216-43a0-806c-2fc39c280e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwvw" Nov 03 21:21:59.064393 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:59.064363 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee9107ea-d216-43a0-806c-2fc39c280e69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ckwvw\" (UID: \"ee9107ea-d216-43a0-806c-2fc39c280e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwvw" Nov 03 21:21:59.066200 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:59.066175 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee9107ea-d216-43a0-806c-2fc39c280e69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ckwvw\" (UID: \"ee9107ea-d216-43a0-806c-2fc39c280e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwvw" Nov 03 21:21:59.124558 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:59.124523 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4vlc\" (UniqueName: \"kubernetes.io/projected/ee9107ea-d216-43a0-806c-2fc39c280e69-kube-api-access-g4vlc\") pod \"maas-api-revoke-sub-3d4156f1-ckwvw\" (UID: \"ee9107ea-d216-43a0-806c-2fc39c280e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwvw" Nov 03 21:21:59.189538 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:59.189504 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwvw" Nov 03 21:21:59.568884 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:59.568860 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwvw"] Nov 03 21:21:59.571078 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:21:59.571047 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee9107ea_d216_43a0_806c_2fc39c280e69.slice/crio-5eebd5137742f69bde8cf41864dea70f8fd4c40baeea7bc6afe513ca85e508e8 WatchSource:0}: Error finding container 5eebd5137742f69bde8cf41864dea70f8fd4c40baeea7bc6afe513ca85e508e8: Status 404 returned error can't find the container with id 5eebd5137742f69bde8cf41864dea70f8fd4c40baeea7bc6afe513ca85e508e8 Nov 03 21:21:59.849900 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:21:59.849801 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwvw" event={"ID":"ee9107ea-d216-43a0-806c-2fc39c280e69","Type":"ContainerStarted","Data":"5eebd5137742f69bde8cf41864dea70f8fd4c40baeea7bc6afe513ca85e508e8"} Nov 03 21:22:00.856553 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:00.856425 4305 generic.go:356] "Generic (PLEG): container finished" podID="ee9107ea-d216-43a0-806c-2fc39c280e69" containerID="d7ffe6678a3dbcc4790d5dfdd4706bb44e99b795ea8670f995e463e78b667597" exitCode=0 Nov 03 21:22:00.856553 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:00.856472 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwvw" event={"ID":"ee9107ea-d216-43a0-806c-2fc39c280e69","Type":"ContainerDied","Data":"d7ffe6678a3dbcc4790d5dfdd4706bb44e99b795ea8670f995e463e78b667597"} Nov 03 21:22:02.001193 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:02.001165 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwvw" Nov 03 21:22:02.094720 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:02.094677 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee9107ea-d216-43a0-806c-2fc39c280e69-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee9107ea-d216-43a0-806c-2fc39c280e69-maas-api-xks-ca\") pod \"ee9107ea-d216-43a0-806c-2fc39c280e69\" (UID: \"ee9107ea-d216-43a0-806c-2fc39c280e69\") " Nov 03 21:22:02.094908 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:02.094782 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee9107ea-d216-43a0-806c-2fc39c280e69-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee9107ea-d216-43a0-806c-2fc39c280e69-maas-api-openshift-ca\") pod \"ee9107ea-d216-43a0-806c-2fc39c280e69\" (UID: \"ee9107ea-d216-43a0-806c-2fc39c280e69\") " Nov 03 21:22:02.094908 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:02.094814 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee9107ea-d216-43a0-806c-2fc39c280e69-kube-api-access-g4vlc\" (UniqueName: \"kubernetes.io/projected/ee9107ea-d216-43a0-806c-2fc39c280e69-kube-api-access-g4vlc\") pod \"ee9107ea-d216-43a0-806c-2fc39c280e69\" (UID: \"ee9107ea-d216-43a0-806c-2fc39c280e69\") " Nov 03 21:22:02.095213 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:02.095180 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee9107ea-d216-43a0-806c-2fc39c280e69-maas-api-openshift-ca" pod "ee9107ea-d216-43a0-806c-2fc39c280e69" (UID: "ee9107ea-d216-43a0-806c-2fc39c280e69"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:22:02.097120 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:02.097093 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee9107ea-d216-43a0-806c-2fc39c280e69-maas-api-xks-ca" pod "ee9107ea-d216-43a0-806c-2fc39c280e69" (UID: "ee9107ea-d216-43a0-806c-2fc39c280e69"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:22:02.097216 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:02.097149 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee9107ea-d216-43a0-806c-2fc39c280e69-kube-api-access-g4vlc" pod "ee9107ea-d216-43a0-806c-2fc39c280e69" (UID: "ee9107ea-d216-43a0-806c-2fc39c280e69"). InnerVolumeSpecName "kube-api-access-g4vlc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:22:02.196478 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:02.196428 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee9107ea-d216-43a0-806c-2fc39c280e69-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:22:02.196478 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:02.196477 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee9107ea-d216-43a0-806c-2fc39c280e69-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:22:02.196691 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:02.196488 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4vlc\" (UniqueName: \"kubernetes.io/projected/ee9107ea-d216-43a0-806c-2fc39c280e69-kube-api-access-g4vlc\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:22:02.867888 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:02.867857 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwvw" Nov 03 21:22:02.867888 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:02.867883 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwvw" event={"ID":"ee9107ea-d216-43a0-806c-2fc39c280e69","Type":"ContainerDied","Data":"5eebd5137742f69bde8cf41864dea70f8fd4c40baeea7bc6afe513ca85e508e8"} Nov 03 21:22:02.868126 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:02.867971 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5eebd5137742f69bde8cf41864dea70f8fd4c40baeea7bc6afe513ca85e508e8" Nov 03 21:22:13.767952 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:13.767853 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:22:16.322557 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:22:16.322522 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:22:24.695032 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:24.694988 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:22:31.616601 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:31.616561 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q"] Nov 03 21:22:31.617095 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:31.617002 4305 cpu_manager.go:414] "containerMap: removing container" podUID="ee9107ea-d216-43a0-806c-2fc39c280e69" containerName="revoke-keys" Nov 03 21:22:31.617095 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:31.617015 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee9107ea-d216-43a0-806c-2fc39c280e69" containerName="revoke-keys" Nov 03 21:22:31.617170 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:31.617101 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee9107ea-d216-43a0-806c-2fc39c280e69" containerName="revoke-keys" Nov 03 21:22:31.620544 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:31.620522 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q" Nov 03 21:22:31.695897 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:31.695867 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-96c0ab-openshift-default-dockercfg-hxjc2\"" Nov 03 21:22:31.712538 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:31.712503 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8ef126d-e530-4343-bb95-1262953e687b-workload-certs\") pod \"e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q\" (UID: \"c8ef126d-e530-4343-bb95-1262953e687b\") " pod="openshift-ingress/e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q" Nov 03 21:22:31.712728 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:31.712630 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8ef126d-e530-4343-bb95-1262953e687b-istio-envoy\") pod \"e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q\" (UID: \"c8ef126d-e530-4343-bb95-1262953e687b\") " pod="openshift-ingress/e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q" Nov 03 21:22:31.712728 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:31.712657 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8ef126d-e530-4343-bb95-1262953e687b-istio-token\") pod \"e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q\" (UID: \"c8ef126d-e530-4343-bb95-1262953e687b\") " pod="openshift-ingress/e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q" Nov 03 21:22:31.712818 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:31.712725 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8ef126d-e530-4343-bb95-1262953e687b-istiod-ca-cert\") pod \"e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q\" (UID: \"c8ef126d-e530-4343-bb95-1262953e687b\") " pod="openshift-ingress/e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q" Nov 03 21:22:31.712818 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:31.712787 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ef126d-e530-4343-bb95-1262953e687b-workload-socket\") pod \"e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q\" (UID: \"c8ef126d-e530-4343-bb95-1262953e687b\") " pod="openshift-ingress/e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q" Nov 03 21:22:31.712818 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:31.712813 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ef126d-e530-4343-bb95-1262953e687b-credential-socket\") pod \"e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q\" (UID: \"c8ef126d-e530-4343-bb95-1262953e687b\") " pod="openshift-ingress/e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q" Nov 03 21:22:31.712939 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:31.712842 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbfg7\" (UniqueName: \"kubernetes.io/projected/c8ef126d-e530-4343-bb95-1262953e687b-kube-api-access-nbfg7\") pod \"e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q\" (UID: \"c8ef126d-e530-4343-bb95-1262953e687b\") " pod="openshift-ingress/e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q" Nov 03 21:22:31.712939 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:31.712917 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8ef126d-e530-4343-bb95-1262953e687b-istio-podinfo\") pod \"e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q\" (UID: \"c8ef126d-e530-4343-bb95-1262953e687b\") " pod="openshift-ingress/e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q" Nov 03 21:22:31.713007 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:31.712960 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8ef126d-e530-4343-bb95-1262953e687b-istio-data\") pod \"e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q\" (UID: \"c8ef126d-e530-4343-bb95-1262953e687b\") " pod="openshift-ingress/e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q" Nov 03 21:22:31.760574 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:31.760544 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q"] Nov 03 21:22:31.813725 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:31.813684 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8ef126d-e530-4343-bb95-1262953e687b-istio-token\") pod \"e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q\" (UID: \"c8ef126d-e530-4343-bb95-1262953e687b\") " pod="openshift-ingress/e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q" Nov 03 21:22:31.813725 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:31.813727 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8ef126d-e530-4343-bb95-1262953e687b-istiod-ca-cert\") pod \"e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q\" (UID: \"c8ef126d-e530-4343-bb95-1262953e687b\") " pod="openshift-ingress/e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q" Nov 03 21:22:31.814032 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:31.813758 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ef126d-e530-4343-bb95-1262953e687b-workload-socket\") pod \"e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q\" (UID: \"c8ef126d-e530-4343-bb95-1262953e687b\") " pod="openshift-ingress/e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q" Nov 03 21:22:31.814032 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:31.813781 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ef126d-e530-4343-bb95-1262953e687b-credential-socket\") pod \"e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q\" (UID: \"c8ef126d-e530-4343-bb95-1262953e687b\") " pod="openshift-ingress/e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q" Nov 03 21:22:31.814032 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:31.813816 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbfg7\" (UniqueName: \"kubernetes.io/projected/c8ef126d-e530-4343-bb95-1262953e687b-kube-api-access-nbfg7\") pod \"e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q\" (UID: \"c8ef126d-e530-4343-bb95-1262953e687b\") " pod="openshift-ingress/e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q" Nov 03 21:22:31.814032 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:31.813851 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8ef126d-e530-4343-bb95-1262953e687b-istio-podinfo\") pod \"e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q\" (UID: \"c8ef126d-e530-4343-bb95-1262953e687b\") " pod="openshift-ingress/e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q" Nov 03 21:22:31.814032 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:31.813891 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8ef126d-e530-4343-bb95-1262953e687b-istio-data\") pod \"e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q\" (UID: \"c8ef126d-e530-4343-bb95-1262953e687b\") " pod="openshift-ingress/e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q" Nov 03 21:22:31.814032 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:31.813937 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8ef126d-e530-4343-bb95-1262953e687b-workload-certs\") pod \"e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q\" (UID: \"c8ef126d-e530-4343-bb95-1262953e687b\") " pod="openshift-ingress/e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q" Nov 03 21:22:31.814032 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:31.814021 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8ef126d-e530-4343-bb95-1262953e687b-istio-envoy\") pod \"e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q\" (UID: \"c8ef126d-e530-4343-bb95-1262953e687b\") " pod="openshift-ingress/e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q" Nov 03 21:22:31.814415 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:31.814245 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ef126d-e530-4343-bb95-1262953e687b-credential-socket\") pod \"e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q\" (UID: \"c8ef126d-e530-4343-bb95-1262953e687b\") " pod="openshift-ingress/e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q" Nov 03 21:22:31.814415 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:31.814280 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ef126d-e530-4343-bb95-1262953e687b-workload-socket\") pod \"e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q\" (UID: \"c8ef126d-e530-4343-bb95-1262953e687b\") " pod="openshift-ingress/e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q" Nov 03 21:22:31.814415 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:31.814401 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8ef126d-e530-4343-bb95-1262953e687b-istio-data\") pod \"e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q\" (UID: \"c8ef126d-e530-4343-bb95-1262953e687b\") " pod="openshift-ingress/e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q" Nov 03 21:22:31.814598 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:31.814485 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8ef126d-e530-4343-bb95-1262953e687b-workload-certs\") pod \"e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q\" (UID: \"c8ef126d-e530-4343-bb95-1262953e687b\") " pod="openshift-ingress/e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q" Nov 03 21:22:31.814710 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:31.814682 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8ef126d-e530-4343-bb95-1262953e687b-istiod-ca-cert\") pod \"e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q\" (UID: \"c8ef126d-e530-4343-bb95-1262953e687b\") " pod="openshift-ingress/e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q" Nov 03 21:22:31.816560 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:31.816535 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8ef126d-e530-4343-bb95-1262953e687b-istio-envoy\") pod \"e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q\" (UID: \"c8ef126d-e530-4343-bb95-1262953e687b\") " pod="openshift-ingress/e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q" Nov 03 21:22:31.816704 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:31.816685 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8ef126d-e530-4343-bb95-1262953e687b-istio-podinfo\") pod \"e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q\" (UID: \"c8ef126d-e530-4343-bb95-1262953e687b\") " pod="openshift-ingress/e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q" Nov 03 21:22:31.879047 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:31.878973 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8ef126d-e530-4343-bb95-1262953e687b-istio-token\") pod \"e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q\" (UID: \"c8ef126d-e530-4343-bb95-1262953e687b\") " pod="openshift-ingress/e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q" Nov 03 21:22:31.879047 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:31.879030 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbfg7\" (UniqueName: \"kubernetes.io/projected/c8ef126d-e530-4343-bb95-1262953e687b-kube-api-access-nbfg7\") pod \"e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q\" (UID: \"c8ef126d-e530-4343-bb95-1262953e687b\") " pod="openshift-ingress/e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q" Nov 03 21:22:31.933405 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:31.933371 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q" Nov 03 21:22:32.365761 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:32.365733 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q"] Nov 03 21:22:32.367246 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:22:32.367219 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8ef126d_e530_4343_bb95_1262953e687b.slice/crio-88e0c3a674123af331b759a0f4b44ea4f9f4b7f4f9e1fd80b8ba337b929ec9c8 WatchSource:0}: Error finding container 88e0c3a674123af331b759a0f4b44ea4f9f4b7f4f9e1fd80b8ba337b929ec9c8: Status 404 returned error can't find the container with id 88e0c3a674123af331b759a0f4b44ea4f9f4b7f4f9e1fd80b8ba337b929ec9c8 Nov 03 21:22:32.369601 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:32.369569 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:22:32.369688 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:32.369635 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:22:32.369688 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:32.369665 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:22:33.027882 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:33.027850 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q" event={"ID":"c8ef126d-e530-4343-bb95-1262953e687b","Type":"ContainerStarted","Data":"fcb9c168f1f2da9713c0daa0fbf4824136e0db88e612006fcf7f3d5663a8fe1a"} Nov 03 21:22:33.028342 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:33.027899 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q" event={"ID":"c8ef126d-e530-4343-bb95-1262953e687b","Type":"ContainerStarted","Data":"88e0c3a674123af331b759a0f4b44ea4f9f4b7f4f9e1fd80b8ba337b929ec9c8"} Nov 03 21:22:33.199495 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:33.199422 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q" podStartSLOduration=2.199405165 podStartE2EDuration="2.199405165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:22:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:22:33.197368675 +0000 UTC m=+2588.422688466" watchObservedRunningTime="2026-11-03 21:22:33.199405165 +0000 UTC m=+2588.424724955" Nov 03 21:22:33.934350 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:33.934299 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q" Nov 03 21:22:33.939959 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:33.939931 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q" Nov 03 21:22:34.032882 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:34.032849 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q" Nov 03 21:22:34.034088 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:34.034066 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q" Nov 03 21:22:39.973201 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:39.973158 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-96c0ab-748d4f7df2n7fk"] Nov 03 21:22:39.977741 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:39.977717 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-96c0ab-748d4f7df2n7fk" Nov 03 21:22:40.034837 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:40.034801 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-96c0ab-748d4f7df2n7fk"] Nov 03 21:22:40.080770 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:40.080733 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-96c0ab\"" Nov 03 21:22:40.080956 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:40.080838 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-e7f85d42-dockercfg-f5r9l\"" Nov 03 21:22:40.107444 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:40.107409 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/165d07de-c15a-413b-83f1-fa25e7b899a1-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-96c0ab-748d4f7df2n7fk\" (UID: \"165d07de-c15a-413b-83f1-fa25e7b899a1\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-96c0ab-748d4f7df2n7fk" Nov 03 21:22:40.107649 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:40.107630 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt9rq\" (UniqueName: \"kubernetes.io/projected/165d07de-c15a-413b-83f1-fa25e7b899a1-kube-api-access-mt9rq\") pod \"payload-processing-e2e-worker-single-main-96c0ab-748d4f7df2n7fk\" (UID: \"165d07de-c15a-413b-83f1-fa25e7b899a1\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-96c0ab-748d4f7df2n7fk" Nov 03 21:22:40.209105 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:40.209062 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt9rq\" (UniqueName: \"kubernetes.io/projected/165d07de-c15a-413b-83f1-fa25e7b899a1-kube-api-access-mt9rq\") pod \"payload-processing-e2e-worker-single-main-96c0ab-748d4f7df2n7fk\" (UID: \"165d07de-c15a-413b-83f1-fa25e7b899a1\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-96c0ab-748d4f7df2n7fk" Nov 03 21:22:40.209333 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:40.209133 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/165d07de-c15a-413b-83f1-fa25e7b899a1-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-96c0ab-748d4f7df2n7fk\" (UID: \"165d07de-c15a-413b-83f1-fa25e7b899a1\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-96c0ab-748d4f7df2n7fk" Nov 03 21:22:40.209933 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:40.209907 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/165d07de-c15a-413b-83f1-fa25e7b899a1-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-96c0ab-748d4f7df2n7fk\" (UID: \"165d07de-c15a-413b-83f1-fa25e7b899a1\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-96c0ab-748d4f7df2n7fk" Nov 03 21:22:40.270287 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:40.270213 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt9rq\" (UniqueName: \"kubernetes.io/projected/165d07de-c15a-413b-83f1-fa25e7b899a1-kube-api-access-mt9rq\") pod \"payload-processing-e2e-worker-single-main-96c0ab-748d4f7df2n7fk\" (UID: \"165d07de-c15a-413b-83f1-fa25e7b899a1\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-96c0ab-748d4f7df2n7fk" Nov 03 21:22:40.294106 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:40.294068 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-96c0ab-748d4f7df2n7fk" Nov 03 21:22:40.671250 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:40.671227 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-96c0ab-748d4f7df2n7fk"] Nov 03 21:22:40.673181 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:22:40.673150 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod165d07de_c15a_413b_83f1_fa25e7b899a1.slice/crio-8e32b2b5394f17dff0b25c369d4f170c18152a0eb47c44f798ef43c5005a8573 WatchSource:0}: Error finding container 8e32b2b5394f17dff0b25c369d4f170c18152a0eb47c44f798ef43c5005a8573: Status 404 returned error can't find the container with id 8e32b2b5394f17dff0b25c369d4f170c18152a0eb47c44f798ef43c5005a8573 Nov 03 21:22:41.077557 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:41.077460 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-96c0ab-748d4f7df2n7fk" event={"ID":"165d07de-c15a-413b-83f1-fa25e7b899a1","Type":"ContainerStarted","Data":"0d19faeb1e1e154ceceff700d34c7bc5bd495ef6cbd3de30d87cd25ed9a1c4b3"} Nov 03 21:22:41.077557 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:41.077525 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-96c0ab-748d4f7df2n7fk" event={"ID":"165d07de-c15a-413b-83f1-fa25e7b899a1","Type":"ContainerStarted","Data":"8e32b2b5394f17dff0b25c369d4f170c18152a0eb47c44f798ef43c5005a8573"} Nov 03 21:22:41.077557 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:41.077550 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-96c0ab-748d4f7df2n7fk" Nov 03 21:22:41.194973 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:41.194914 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-96c0ab-748d4f7df2n7fk" podStartSLOduration=2.194898622 podStartE2EDuration="2.194898622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:22:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:22:41.19186445 +0000 UTC m=+2596.417184259" watchObservedRunningTime="2026-11-03 21:22:41.194898622 +0000 UTC m=+2596.420218413" Nov 03 21:22:46.322712 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:22:46.322673 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:22:50.759023 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:50.758988 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:22:52.087006 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:22:52.086976 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-96c0ab-748d4f7df2n7fk" Nov 03 21:23:01.216633 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:23:01.216597 4305 generic.go:356] "Generic (PLEG): container finished" podID="c8ef126d-e530-4343-bb95-1262953e687b" containerID="fcb9c168f1f2da9713c0daa0fbf4824136e0db88e612006fcf7f3d5663a8fe1a" exitCode=0 Nov 03 21:23:01.217078 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:23:01.216673 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q" event={"ID":"c8ef126d-e530-4343-bb95-1262953e687b","Type":"ContainerDied","Data":"fcb9c168f1f2da9713c0daa0fbf4824136e0db88e612006fcf7f3d5663a8fe1a"} Nov 03 21:23:01.217128 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:23:01.217092 4305 scope.go:122] "RemoveContainer" containerID="fcb9c168f1f2da9713c0daa0fbf4824136e0db88e612006fcf7f3d5663a8fe1a" Nov 03 21:23:01.218372 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:23:01.218345 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:23:01.218450 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:23:01.218410 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:23:01.218503 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:23:01.218460 4305 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Nov 03 21:23:02.224309 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:23:02.224275 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q" event={"ID":"c8ef126d-e530-4343-bb95-1262953e687b","Type":"ContainerStarted","Data":"cc44208ffaaf91b25786e283c69a6c6af6341c5bdc4445081f45251ec013a59e"} Nov 03 21:23:02.934598 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:23:02.934557 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q" Nov 03 21:23:02.935882 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:23:02.935856 4305 patch_prober.go:29] interesting pod/e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body= Nov 03 21:23:02.935967 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:23:02.935909 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q" podUID="c8ef126d-e530-4343-bb95-1262953e687b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" Nov 03 21:23:03.934825 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:23:03.934780 4305 patch_prober.go:29] interesting pod/e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body= Nov 03 21:23:03.935364 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:23:03.934844 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q" podUID="c8ef126d-e530-4343-bb95-1262953e687b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" Nov 03 21:23:04.033497 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:23:04.033461 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q" Nov 03 21:23:04.934605 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:23:04.934568 4305 patch_prober.go:29] interesting pod/e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body= Nov 03 21:23:04.934819 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:23:04.934620 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q" podUID="c8ef126d-e530-4343-bb95-1262953e687b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" Nov 03 21:23:05.934699 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:23:05.934657 4305 patch_prober.go:29] interesting pod/e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body= Nov 03 21:23:05.935148 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:23:05.934721 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q" podUID="c8ef126d-e530-4343-bb95-1262953e687b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" Nov 03 21:23:06.934181 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:23:06.934144 4305 patch_prober.go:29] interesting pod/e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body= Nov 03 21:23:06.934384 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:23:06.934202 4305 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q" podUID="c8ef126d-e530-4343-bb95-1262953e687b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" Nov 03 21:23:07.938464 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:23:07.938405 4305 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q" Nov 03 21:23:07.939785 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:23:07.939764 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q" Nov 03 21:23:16.322798 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:23:16.322759 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:23:19.493667 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:23:19.493630 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:23:27.048090 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:23:27.048058 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mlgs7"] Nov 03 21:23:27.096101 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:23:27.096064 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mlgs7"] Nov 03 21:23:28.894910 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:23:28.894872 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef0ad8a6-0850-46fe-8e2b-5069bf15835a" path="/var/lib/kubelet/pods/ef0ad8a6-0850-46fe-8e2b-5069bf15835a/volumes" Nov 03 21:23:46.322812 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:23:46.322726 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:23:47.179102 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:23:47.179072 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:23:55.049812 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:23:55.049777 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-smcsd"] Nov 03 21:23:55.099465 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:23:55.099378 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-smcsd"] Nov 03 21:23:56.894248 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:23:56.894214 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f08548e-3650-4b24-84e9-1ae1403ab828" path="/var/lib/kubelet/pods/8f08548e-3650-4b24-84e9-1ae1403ab828/volumes" Nov 03 21:24:01.337906 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:01.337875 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-189e7f0a-lqv78"] Nov 03 21:24:01.341931 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:01.341909 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-189e7f0a-lqv78" Nov 03 21:24:01.392081 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:01.392052 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b9c18a0-ad8f-49ed-a337-2128333f6332-maas-api-xks-ca\") pod \"maas-api-revoke-sub-189e7f0a-lqv78\" (UID: \"5b9c18a0-ad8f-49ed-a337-2128333f6332\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-189e7f0a-lqv78" Nov 03 21:24:01.392272 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:01.392100 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b9c18a0-ad8f-49ed-a337-2128333f6332-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-189e7f0a-lqv78\" (UID: \"5b9c18a0-ad8f-49ed-a337-2128333f6332\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-189e7f0a-lqv78" Nov 03 21:24:01.392318 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:01.392287 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7h6l\" (UniqueName: \"kubernetes.io/projected/5b9c18a0-ad8f-49ed-a337-2128333f6332-kube-api-access-m7h6l\") pod \"maas-api-revoke-sub-189e7f0a-lqv78\" (UID: \"5b9c18a0-ad8f-49ed-a337-2128333f6332\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-189e7f0a-lqv78" Nov 03 21:24:01.399823 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:01.399795 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-189e7f0a-lqv78"] Nov 03 21:24:01.446616 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:01.446537 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:24:01.446616 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:01.446572 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:24:01.446859 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:01.446624 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:24:01.446859 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:01.446658 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77vtd\"" Nov 03 21:24:01.493657 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:01.493621 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b9c18a0-ad8f-49ed-a337-2128333f6332-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-189e7f0a-lqv78\" (UID: \"5b9c18a0-ad8f-49ed-a337-2128333f6332\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-189e7f0a-lqv78" Nov 03 21:24:01.493864 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:01.493737 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7h6l\" (UniqueName: \"kubernetes.io/projected/5b9c18a0-ad8f-49ed-a337-2128333f6332-kube-api-access-m7h6l\") pod \"maas-api-revoke-sub-189e7f0a-lqv78\" (UID: \"5b9c18a0-ad8f-49ed-a337-2128333f6332\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-189e7f0a-lqv78" Nov 03 21:24:01.493864 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:01.493785 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b9c18a0-ad8f-49ed-a337-2128333f6332-maas-api-xks-ca\") pod \"maas-api-revoke-sub-189e7f0a-lqv78\" (UID: \"5b9c18a0-ad8f-49ed-a337-2128333f6332\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-189e7f0a-lqv78" Nov 03 21:24:01.494290 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:01.494268 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b9c18a0-ad8f-49ed-a337-2128333f6332-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-189e7f0a-lqv78\" (UID: \"5b9c18a0-ad8f-49ed-a337-2128333f6332\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-189e7f0a-lqv78" Nov 03 21:24:01.496535 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:01.496509 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b9c18a0-ad8f-49ed-a337-2128333f6332-maas-api-xks-ca\") pod \"maas-api-revoke-sub-189e7f0a-lqv78\" (UID: \"5b9c18a0-ad8f-49ed-a337-2128333f6332\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-189e7f0a-lqv78" Nov 03 21:24:01.555499 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:01.555465 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7h6l\" (UniqueName: \"kubernetes.io/projected/5b9c18a0-ad8f-49ed-a337-2128333f6332-kube-api-access-m7h6l\") pod \"maas-api-revoke-sub-189e7f0a-lqv78\" (UID: \"5b9c18a0-ad8f-49ed-a337-2128333f6332\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-189e7f0a-lqv78" Nov 03 21:24:01.655820 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:01.655777 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-189e7f0a-lqv78" Nov 03 21:24:02.033917 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:02.033890 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-189e7f0a-lqv78"] Nov 03 21:24:02.035889 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:24:02.035855 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b9c18a0_ad8f_49ed_a337_2128333f6332.slice/crio-bf5e6439a69d4a16682f7db50b7e14e0a77bc3079282491ee63befcaedfadf48 WatchSource:0}: Error finding container bf5e6439a69d4a16682f7db50b7e14e0a77bc3079282491ee63befcaedfadf48: Status 404 returned error can't find the container with id bf5e6439a69d4a16682f7db50b7e14e0a77bc3079282491ee63befcaedfadf48 Nov 03 21:24:02.544409 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:02.544377 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-189e7f0a-lqv78" event={"ID":"5b9c18a0-ad8f-49ed-a337-2128333f6332","Type":"ContainerStarted","Data":"029dfc716524daf222976cb611a89fc8e6e7330d2e33bc5bb8635728fb76a2fc"} Nov 03 21:24:02.544945 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:02.544427 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-189e7f0a-lqv78" event={"ID":"5b9c18a0-ad8f-49ed-a337-2128333f6332","Type":"ContainerStarted","Data":"bf5e6439a69d4a16682f7db50b7e14e0a77bc3079282491ee63befcaedfadf48"} Nov 03 21:24:02.672921 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:02.672857 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-189e7f0a-lqv78" podStartSLOduration=1.672842568 podStartE2EDuration="1.672842568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:24:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:24:02.669641994 +0000 UTC m=+2677.894961798" watchObservedRunningTime="2026-11-03 21:24:02.672842568 +0000 UTC m=+2677.898162359" Nov 03 21:24:03.550872 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:03.550836 4305 generic.go:356] "Generic (PLEG): container finished" podID="5b9c18a0-ad8f-49ed-a337-2128333f6332" containerID="029dfc716524daf222976cb611a89fc8e6e7330d2e33bc5bb8635728fb76a2fc" exitCode=0 Nov 03 21:24:03.551298 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:03.550920 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-189e7f0a-lqv78" event={"ID":"5b9c18a0-ad8f-49ed-a337-2128333f6332","Type":"ContainerDied","Data":"029dfc716524daf222976cb611a89fc8e6e7330d2e33bc5bb8635728fb76a2fc"} Nov 03 21:24:04.705216 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:04.705191 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-189e7f0a-lqv78" Nov 03 21:24:04.826987 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:04.826886 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b9c18a0-ad8f-49ed-a337-2128333f6332-kube-api-access-m7h6l\" (UniqueName: \"kubernetes.io/projected/5b9c18a0-ad8f-49ed-a337-2128333f6332-kube-api-access-m7h6l\") pod \"5b9c18a0-ad8f-49ed-a337-2128333f6332\" (UID: \"5b9c18a0-ad8f-49ed-a337-2128333f6332\") " Nov 03 21:24:04.826987 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:04.826934 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b9c18a0-ad8f-49ed-a337-2128333f6332-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b9c18a0-ad8f-49ed-a337-2128333f6332-maas-api-xks-ca\") pod \"5b9c18a0-ad8f-49ed-a337-2128333f6332\" (UID: \"5b9c18a0-ad8f-49ed-a337-2128333f6332\") " Nov 03 21:24:04.826987 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:04.826972 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b9c18a0-ad8f-49ed-a337-2128333f6332-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b9c18a0-ad8f-49ed-a337-2128333f6332-maas-api-openshift-ca\") pod \"5b9c18a0-ad8f-49ed-a337-2128333f6332\" (UID: \"5b9c18a0-ad8f-49ed-a337-2128333f6332\") " Nov 03 21:24:04.828300 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:04.828271 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b9c18a0-ad8f-49ed-a337-2128333f6332-maas-api-openshift-ca" pod "5b9c18a0-ad8f-49ed-a337-2128333f6332" (UID: "5b9c18a0-ad8f-49ed-a337-2128333f6332"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:24:04.830370 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:04.830310 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b9c18a0-ad8f-49ed-a337-2128333f6332-kube-api-access-m7h6l" pod "5b9c18a0-ad8f-49ed-a337-2128333f6332" (UID: "5b9c18a0-ad8f-49ed-a337-2128333f6332"). InnerVolumeSpecName "kube-api-access-m7h6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:24:04.831830 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:04.831806 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b9c18a0-ad8f-49ed-a337-2128333f6332-maas-api-xks-ca" pod "5b9c18a0-ad8f-49ed-a337-2128333f6332" (UID: "5b9c18a0-ad8f-49ed-a337-2128333f6332"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:24:04.928477 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:04.928448 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7h6l\" (UniqueName: \"kubernetes.io/projected/5b9c18a0-ad8f-49ed-a337-2128333f6332-kube-api-access-m7h6l\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:24:04.928477 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:04.928475 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b9c18a0-ad8f-49ed-a337-2128333f6332-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:24:04.928667 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:04.928485 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b9c18a0-ad8f-49ed-a337-2128333f6332-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:24:05.568503 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:05.568478 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-189e7f0a-lqv78" Nov 03 21:24:05.568689 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:05.568470 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-189e7f0a-lqv78" event={"ID":"5b9c18a0-ad8f-49ed-a337-2128333f6332","Type":"ContainerDied","Data":"bf5e6439a69d4a16682f7db50b7e14e0a77bc3079282491ee63befcaedfadf48"} Nov 03 21:24:05.568689 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:05.568623 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf5e6439a69d4a16682f7db50b7e14e0a77bc3079282491ee63befcaedfadf48" Nov 03 21:24:13.430394 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:13.430355 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-96c0ab-ggmdr"] Nov 03 21:24:13.430909 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:13.430877 4305 cpu_manager.go:414] "containerMap: removing container" podUID="5b9c18a0-ad8f-49ed-a337-2128333f6332" containerName="revoke-keys" Nov 03 21:24:13.430909 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:13.430891 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b9c18a0-ad8f-49ed-a337-2128333f6332" containerName="revoke-keys" Nov 03 21:24:13.431004 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:13.430992 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b9c18a0-ad8f-49ed-a337-2128333f6332" containerName="revoke-keys" Nov 03 21:24:13.434200 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:13.434182 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-96c0ab-ggmdr" Nov 03 21:24:13.488682 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:13.488646 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:24:13.522149 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:13.522106 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzqpp\" (UniqueName: \"kubernetes.io/projected/1ada3c5d-955d-4d5f-8d33-db207ddb6c92-kube-api-access-vzqpp\") pod \"maas-api-revoke-keys-e2e-worker-single-main-96c0ab-ggmdr\" (UID: \"1ada3c5d-955d-4d5f-8d33-db207ddb6c92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-96c0ab-ggmdr" Nov 03 21:24:13.522386 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:13.522198 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ada3c5d-955d-4d5f-8d33-db207ddb6c92-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-96c0ab-ggmdr\" (UID: \"1ada3c5d-955d-4d5f-8d33-db207ddb6c92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-96c0ab-ggmdr" Nov 03 21:24:13.522386 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:13.522241 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ada3c5d-955d-4d5f-8d33-db207ddb6c92-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-96c0ab-ggmdr\" (UID: \"1ada3c5d-955d-4d5f-8d33-db207ddb6c92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-96c0ab-ggmdr" Nov 03 21:24:13.538186 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:13.538152 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:24:13.538382 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:13.538206 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:24:13.538382 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:13.538214 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77vtd\"" Nov 03 21:24:13.540606 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:13.540586 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-96c0ab-ggmdr"] Nov 03 21:24:13.623917 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:13.623872 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzqpp\" (UniqueName: \"kubernetes.io/projected/1ada3c5d-955d-4d5f-8d33-db207ddb6c92-kube-api-access-vzqpp\") pod \"maas-api-revoke-keys-e2e-worker-single-main-96c0ab-ggmdr\" (UID: \"1ada3c5d-955d-4d5f-8d33-db207ddb6c92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-96c0ab-ggmdr" Nov 03 21:24:13.623917 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:13.623919 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ada3c5d-955d-4d5f-8d33-db207ddb6c92-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-96c0ab-ggmdr\" (UID: \"1ada3c5d-955d-4d5f-8d33-db207ddb6c92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-96c0ab-ggmdr" Nov 03 21:24:13.624163 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:13.623960 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ada3c5d-955d-4d5f-8d33-db207ddb6c92-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-96c0ab-ggmdr\" (UID: \"1ada3c5d-955d-4d5f-8d33-db207ddb6c92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-96c0ab-ggmdr" Nov 03 21:24:13.624674 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:13.624654 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ada3c5d-955d-4d5f-8d33-db207ddb6c92-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-96c0ab-ggmdr\" (UID: \"1ada3c5d-955d-4d5f-8d33-db207ddb6c92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-96c0ab-ggmdr" Nov 03 21:24:13.626563 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:13.626531 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ada3c5d-955d-4d5f-8d33-db207ddb6c92-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-96c0ab-ggmdr\" (UID: \"1ada3c5d-955d-4d5f-8d33-db207ddb6c92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-96c0ab-ggmdr" Nov 03 21:24:13.685342 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:13.685257 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzqpp\" (UniqueName: \"kubernetes.io/projected/1ada3c5d-955d-4d5f-8d33-db207ddb6c92-kube-api-access-vzqpp\") pod \"maas-api-revoke-keys-e2e-worker-single-main-96c0ab-ggmdr\" (UID: \"1ada3c5d-955d-4d5f-8d33-db207ddb6c92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-96c0ab-ggmdr" Nov 03 21:24:13.747996 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:13.747969 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-96c0ab-ggmdr" Nov 03 21:24:14.130746 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:14.130700 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-96c0ab-ggmdr"] Nov 03 21:24:14.130901 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:24:14.130856 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ada3c5d_955d_4d5f_8d33_db207ddb6c92.slice/crio-af78fc28eeaadd2e970d0271a2d3a27221ef97e94a02e3eef94704168f26d0ab WatchSource:0}: Error finding container af78fc28eeaadd2e970d0271a2d3a27221ef97e94a02e3eef94704168f26d0ab: Status 404 returned error can't find the container with id af78fc28eeaadd2e970d0271a2d3a27221ef97e94a02e3eef94704168f26d0ab Nov 03 21:24:14.618952 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:14.618918 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-96c0ab-ggmdr" event={"ID":"1ada3c5d-955d-4d5f-8d33-db207ddb6c92","Type":"ContainerStarted","Data":"e331d50421df6d7796da266446a1f2383de18d263c09c3dccc380399eef75a4e"} Nov 03 21:24:14.619470 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:14.618980 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-96c0ab-ggmdr" event={"ID":"1ada3c5d-955d-4d5f-8d33-db207ddb6c92","Type":"ContainerStarted","Data":"af78fc28eeaadd2e970d0271a2d3a27221ef97e94a02e3eef94704168f26d0ab"} Nov 03 21:24:14.734948 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:14.734831 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-96c0ab-ggmdr" podStartSLOduration=1.73481503 podStartE2EDuration="1.73481503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:24:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:24:14.73366093 +0000 UTC m=+2689.958980746" watchObservedRunningTime="2026-11-03 21:24:14.73481503 +0000 UTC m=+2689.960134821" Nov 03 21:24:15.625749 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:15.625713 4305 generic.go:356] "Generic (PLEG): container finished" podID="1ada3c5d-955d-4d5f-8d33-db207ddb6c92" containerID="e331d50421df6d7796da266446a1f2383de18d263c09c3dccc380399eef75a4e" exitCode=0 Nov 03 21:24:15.626191 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:15.625796 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-96c0ab-ggmdr" event={"ID":"1ada3c5d-955d-4d5f-8d33-db207ddb6c92","Type":"ContainerDied","Data":"e331d50421df6d7796da266446a1f2383de18d263c09c3dccc380399eef75a4e"} Nov 03 21:24:16.328996 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:16.328958 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:24:16.349146 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:24:16.349107 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:24:16.779273 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:16.779243 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-96c0ab-ggmdr" Nov 03 21:24:16.962145 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:16.962107 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ada3c5d-955d-4d5f-8d33-db207ddb6c92-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ada3c5d-955d-4d5f-8d33-db207ddb6c92-maas-api-xks-ca\") pod \"1ada3c5d-955d-4d5f-8d33-db207ddb6c92\" (UID: \"1ada3c5d-955d-4d5f-8d33-db207ddb6c92\") " Nov 03 21:24:16.962372 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:16.962165 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ada3c5d-955d-4d5f-8d33-db207ddb6c92-kube-api-access-vzqpp\" (UniqueName: \"kubernetes.io/projected/1ada3c5d-955d-4d5f-8d33-db207ddb6c92-kube-api-access-vzqpp\") pod \"1ada3c5d-955d-4d5f-8d33-db207ddb6c92\" (UID: \"1ada3c5d-955d-4d5f-8d33-db207ddb6c92\") " Nov 03 21:24:16.962372 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:16.962246 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ada3c5d-955d-4d5f-8d33-db207ddb6c92-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ada3c5d-955d-4d5f-8d33-db207ddb6c92-maas-api-openshift-ca\") pod \"1ada3c5d-955d-4d5f-8d33-db207ddb6c92\" (UID: \"1ada3c5d-955d-4d5f-8d33-db207ddb6c92\") " Nov 03 21:24:16.962703 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:16.962678 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ada3c5d-955d-4d5f-8d33-db207ddb6c92-maas-api-openshift-ca" pod "1ada3c5d-955d-4d5f-8d33-db207ddb6c92" (UID: "1ada3c5d-955d-4d5f-8d33-db207ddb6c92"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:24:16.964402 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:16.964368 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ada3c5d-955d-4d5f-8d33-db207ddb6c92-maas-api-xks-ca" pod "1ada3c5d-955d-4d5f-8d33-db207ddb6c92" (UID: "1ada3c5d-955d-4d5f-8d33-db207ddb6c92"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:24:16.964530 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:16.964476 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ada3c5d-955d-4d5f-8d33-db207ddb6c92-kube-api-access-vzqpp" pod "1ada3c5d-955d-4d5f-8d33-db207ddb6c92" (UID: "1ada3c5d-955d-4d5f-8d33-db207ddb6c92"). InnerVolumeSpecName "kube-api-access-vzqpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:24:17.064056 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:17.064013 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ada3c5d-955d-4d5f-8d33-db207ddb6c92-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:24:17.064056 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:17.064049 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzqpp\" (UniqueName: \"kubernetes.io/projected/1ada3c5d-955d-4d5f-8d33-db207ddb6c92-kube-api-access-vzqpp\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:24:17.064056 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:17.064060 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ada3c5d-955d-4d5f-8d33-db207ddb6c92-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:24:17.638305 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:17.638270 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-96c0ab-ggmdr" event={"ID":"1ada3c5d-955d-4d5f-8d33-db207ddb6c92","Type":"ContainerDied","Data":"af78fc28eeaadd2e970d0271a2d3a27221ef97e94a02e3eef94704168f26d0ab"} Nov 03 21:24:17.638305 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:17.638303 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-96c0ab-ggmdr" Nov 03 21:24:17.638569 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:17.638351 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af78fc28eeaadd2e970d0271a2d3a27221ef97e94a02e3eef94704168f26d0ab" Nov 03 21:24:24.939589 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:24.939546 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-96c0ab-748d4f7df2n7fk"] Nov 03 21:24:26.069583 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:26.069538 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vscbw"] Nov 03 21:24:26.072124 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:26.072095 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vscbw"] Nov 03 21:24:26.894350 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:26.894316 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f19c63e0-0d0c-4b63-a59f-7aca9057efc2" path="/var/lib/kubelet/pods/f19c63e0-0d0c-4b63-a59f-7aca9057efc2/volumes" Nov 03 21:24:29.940517 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:29.940475 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-96c0ab-748d4f7df2n7fk" podUID="165d07de-c15a-413b-83f1-fa25e7b899a1" containerName="payload-processing" containerID="cri-o://0d19faeb1e1e154ceceff700d34c7bc5bd495ef6cbd3de30d87cd25ed9a1c4b3" gracePeriod=25 Nov 03 21:24:30.554613 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:30.554579 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-96c0ab-ggmdr"] Nov 03 21:24:30.604174 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:30.604137 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-96c0ab-ggmdr"] Nov 03 21:24:30.893945 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:30.893915 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ada3c5d-955d-4d5f-8d33-db207ddb6c92" path="/var/lib/kubelet/pods/1ada3c5d-955d-4d5f-8d33-db207ddb6c92/volumes" Nov 03 21:24:32.084989 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:32.084951 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-96c0ab-748d4f7df2n7fk" podUID="165d07de-c15a-413b-83f1-fa25e7b899a1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:24:32.877006 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:32.876974 4305 scope.go:122] "RemoveContainer" containerID="1759a8098b58453253ed8acd4b1fd82577ff6215780cf8b7fb619f7e79ee54a7" Nov 03 21:24:33.140716 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:33.140616 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:24:37.836234 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:37.836203 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q"] Nov 03 21:24:37.836710 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:37.836395 4305 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q" podUID="c8ef126d-e530-4343-bb95-1262953e687b" containerName="istio-proxy" containerID="cri-o://cc44208ffaaf91b25786e283c69a6c6af6341c5bdc4445081f45251ec013a59e" gracePeriod=30 Nov 03 21:24:37.941628 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:37.941588 4305 patch_prober.go:29] interesting pod/e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:24:37.941811 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:37.941651 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q" podUID="c8ef126d-e530-4343-bb95-1262953e687b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:24:39.335039 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:39.334998 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwvw"] Nov 03 21:24:39.384375 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:39.384341 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ckwvw"] Nov 03 21:24:39.456504 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:39.456474 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wxc9q"] Nov 03 21:24:39.456976 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:39.456961 4305 cpu_manager.go:414] "containerMap: removing container" podUID="1ada3c5d-955d-4d5f-8d33-db207ddb6c92" containerName="revoke-keys" Nov 03 21:24:39.457036 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:39.456977 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ada3c5d-955d-4d5f-8d33-db207ddb6c92" containerName="revoke-keys" Nov 03 21:24:39.457073 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:39.457051 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ada3c5d-955d-4d5f-8d33-db207ddb6c92" containerName="revoke-keys" Nov 03 21:24:39.465084 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:39.463509 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wxc9q" Nov 03 21:24:39.517111 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:39.517072 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:24:39.517111 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:39.517092 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:24:39.517401 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:39.517157 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-77vtd\"" Nov 03 21:24:39.517401 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:39.517278 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:24:39.517401 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:39.517292 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wxc9q"] Nov 03 21:24:39.615575 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:39.615480 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7172d4b-d0b8-4090-a957-07521eb40205-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wxc9q\" (UID: \"b7172d4b-d0b8-4090-a957-07521eb40205\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wxc9q" Nov 03 21:24:39.615575 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:39.615541 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhk8s\" (UniqueName: \"kubernetes.io/projected/b7172d4b-d0b8-4090-a957-07521eb40205-kube-api-access-lhk8s\") pod \"maas-api-revoke-sub-3d4156f1-wxc9q\" (UID: \"b7172d4b-d0b8-4090-a957-07521eb40205\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wxc9q" Nov 03 21:24:39.615796 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:39.615682 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7172d4b-d0b8-4090-a957-07521eb40205-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wxc9q\" (UID: \"b7172d4b-d0b8-4090-a957-07521eb40205\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wxc9q" Nov 03 21:24:39.716823 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:39.716789 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7172d4b-d0b8-4090-a957-07521eb40205-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wxc9q\" (UID: \"b7172d4b-d0b8-4090-a957-07521eb40205\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wxc9q" Nov 03 21:24:39.717022 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:39.716939 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7172d4b-d0b8-4090-a957-07521eb40205-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wxc9q\" (UID: \"b7172d4b-d0b8-4090-a957-07521eb40205\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wxc9q" Nov 03 21:24:39.717022 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:39.716999 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhk8s\" (UniqueName: \"kubernetes.io/projected/b7172d4b-d0b8-4090-a957-07521eb40205-kube-api-access-lhk8s\") pod \"maas-api-revoke-sub-3d4156f1-wxc9q\" (UID: \"b7172d4b-d0b8-4090-a957-07521eb40205\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wxc9q" Nov 03 21:24:39.717759 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:39.717731 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7172d4b-d0b8-4090-a957-07521eb40205-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wxc9q\" (UID: \"b7172d4b-d0b8-4090-a957-07521eb40205\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wxc9q" Nov 03 21:24:39.719359 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:39.719339 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7172d4b-d0b8-4090-a957-07521eb40205-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wxc9q\" (UID: \"b7172d4b-d0b8-4090-a957-07521eb40205\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wxc9q" Nov 03 21:24:39.778178 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:39.778144 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhk8s\" (UniqueName: \"kubernetes.io/projected/b7172d4b-d0b8-4090-a957-07521eb40205-kube-api-access-lhk8s\") pod \"maas-api-revoke-sub-3d4156f1-wxc9q\" (UID: \"b7172d4b-d0b8-4090-a957-07521eb40205\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wxc9q" Nov 03 21:24:40.077465 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:40.077402 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wxc9q" Nov 03 21:24:40.452628 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:40.452600 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wxc9q"] Nov 03 21:24:40.454313 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:24:40.454283 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7172d4b_d0b8_4090_a957_07521eb40205.slice/crio-5900b887125941cf2946605f9d3b3e121ec2f85208561227ea8945247b391f11 WatchSource:0}: Error finding container 5900b887125941cf2946605f9d3b3e121ec2f85208561227ea8945247b391f11: Status 404 returned error can't find the container with id 5900b887125941cf2946605f9d3b3e121ec2f85208561227ea8945247b391f11 Nov 03 21:24:40.764112 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:40.764082 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wxc9q" event={"ID":"b7172d4b-d0b8-4090-a957-07521eb40205","Type":"ContainerStarted","Data":"5900b887125941cf2946605f9d3b3e121ec2f85208561227ea8945247b391f11"} Nov 03 21:24:40.894121 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:40.894089 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee9107ea-d216-43a0-806c-2fc39c280e69" path="/var/lib/kubelet/pods/ee9107ea-d216-43a0-806c-2fc39c280e69/volumes" Nov 03 21:24:41.771210 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:41.771176 4305 generic.go:356] "Generic (PLEG): container finished" podID="b7172d4b-d0b8-4090-a957-07521eb40205" containerID="9b528d3beb3d581c84ceab8aa8e3095327dbe4e1ccb9c9070261f6941bdd65e4" exitCode=0 Nov 03 21:24:41.771678 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:41.771237 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wxc9q" event={"ID":"b7172d4b-d0b8-4090-a957-07521eb40205","Type":"ContainerDied","Data":"9b528d3beb3d581c84ceab8aa8e3095327dbe4e1ccb9c9070261f6941bdd65e4"} Nov 03 21:24:42.084718 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:42.084627 4305 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-96c0ab-748d4f7df2n7fk" podUID="165d07de-c15a-413b-83f1-fa25e7b899a1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:24:42.962960 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:42.962936 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wxc9q" Nov 03 21:24:43.053663 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:43.053577 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7172d4b-d0b8-4090-a957-07521eb40205-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7172d4b-d0b8-4090-a957-07521eb40205-maas-api-openshift-ca\") pod \"b7172d4b-d0b8-4090-a957-07521eb40205\" (UID: \"b7172d4b-d0b8-4090-a957-07521eb40205\") " Nov 03 21:24:43.053663 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:43.053645 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7172d4b-d0b8-4090-a957-07521eb40205-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7172d4b-d0b8-4090-a957-07521eb40205-maas-api-xks-ca\") pod \"b7172d4b-d0b8-4090-a957-07521eb40205\" (UID: \"b7172d4b-d0b8-4090-a957-07521eb40205\") " Nov 03 21:24:43.053910 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:43.053736 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7172d4b-d0b8-4090-a957-07521eb40205-kube-api-access-lhk8s\" (UniqueName: \"kubernetes.io/projected/b7172d4b-d0b8-4090-a957-07521eb40205-kube-api-access-lhk8s\") pod \"b7172d4b-d0b8-4090-a957-07521eb40205\" (UID: \"b7172d4b-d0b8-4090-a957-07521eb40205\") " Nov 03 21:24:43.054604 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:43.054013 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7172d4b-d0b8-4090-a957-07521eb40205-maas-api-openshift-ca" pod "b7172d4b-d0b8-4090-a957-07521eb40205" (UID: "b7172d4b-d0b8-4090-a957-07521eb40205"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:24:43.054604 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:43.054262 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7172d4b-d0b8-4090-a957-07521eb40205-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:24:43.056898 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:43.056865 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7172d4b-d0b8-4090-a957-07521eb40205-kube-api-access-lhk8s" pod "b7172d4b-d0b8-4090-a957-07521eb40205" (UID: "b7172d4b-d0b8-4090-a957-07521eb40205"). InnerVolumeSpecName "kube-api-access-lhk8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:24:43.056898 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:43.056875 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7172d4b-d0b8-4090-a957-07521eb40205-maas-api-xks-ca" pod "b7172d4b-d0b8-4090-a957-07521eb40205" (UID: "b7172d4b-d0b8-4090-a957-07521eb40205"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:24:43.091490 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:43.091459 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q" Nov 03 21:24:43.155001 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:43.154971 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7172d4b-d0b8-4090-a957-07521eb40205-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:24:43.155001 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:43.154996 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhk8s\" (UniqueName: \"kubernetes.io/projected/b7172d4b-d0b8-4090-a957-07521eb40205-kube-api-access-lhk8s\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:24:43.256256 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:43.256224 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8ef126d-e530-4343-bb95-1262953e687b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8ef126d-e530-4343-bb95-1262953e687b-workload-certs\") pod \"c8ef126d-e530-4343-bb95-1262953e687b\" (UID: \"c8ef126d-e530-4343-bb95-1262953e687b\") " Nov 03 21:24:43.256450 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:43.256297 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8ef126d-e530-4343-bb95-1262953e687b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8ef126d-e530-4343-bb95-1262953e687b-istio-envoy\") pod \"c8ef126d-e530-4343-bb95-1262953e687b\" (UID: \"c8ef126d-e530-4343-bb95-1262953e687b\") " Nov 03 21:24:43.256450 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:43.256329 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c8ef126d-e530-4343-bb95-1262953e687b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8ef126d-e530-4343-bb95-1262953e687b-istio-podinfo\") pod \"c8ef126d-e530-4343-bb95-1262953e687b\" (UID: \"c8ef126d-e530-4343-bb95-1262953e687b\") " Nov 03 21:24:43.256450 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:43.256395 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8ef126d-e530-4343-bb95-1262953e687b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8ef126d-e530-4343-bb95-1262953e687b-istio-data\") pod \"c8ef126d-e530-4343-bb95-1262953e687b\" (UID: \"c8ef126d-e530-4343-bb95-1262953e687b\") " Nov 03 21:24:43.256634 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:43.256529 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8ef126d-e530-4343-bb95-1262953e687b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ef126d-e530-4343-bb95-1262953e687b-workload-socket\") pod \"c8ef126d-e530-4343-bb95-1262953e687b\" (UID: \"c8ef126d-e530-4343-bb95-1262953e687b\") " Nov 03 21:24:43.256634 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:43.256573 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8ef126d-e530-4343-bb95-1262953e687b-istio-token\" (UniqueName: \"kubernetes.io/projected/c8ef126d-e530-4343-bb95-1262953e687b-istio-token\") pod \"c8ef126d-e530-4343-bb95-1262953e687b\" (UID: \"c8ef126d-e530-4343-bb95-1262953e687b\") " Nov 03 21:24:43.256634 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:43.256575 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8ef126d-e530-4343-bb95-1262953e687b-workload-certs" pod "c8ef126d-e530-4343-bb95-1262953e687b" (UID: "c8ef126d-e530-4343-bb95-1262953e687b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:24:43.256634 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:43.256606 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8ef126d-e530-4343-bb95-1262953e687b-kube-api-access-nbfg7\" (UniqueName: \"kubernetes.io/projected/c8ef126d-e530-4343-bb95-1262953e687b-kube-api-access-nbfg7\") pod \"c8ef126d-e530-4343-bb95-1262953e687b\" (UID: \"c8ef126d-e530-4343-bb95-1262953e687b\") " Nov 03 21:24:43.256850 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:43.256662 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8ef126d-e530-4343-bb95-1262953e687b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ef126d-e530-4343-bb95-1262953e687b-credential-socket\") pod \"c8ef126d-e530-4343-bb95-1262953e687b\" (UID: \"c8ef126d-e530-4343-bb95-1262953e687b\") " Nov 03 21:24:43.256850 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:43.256729 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8ef126d-e530-4343-bb95-1262953e687b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8ef126d-e530-4343-bb95-1262953e687b-istiod-ca-cert\") pod \"c8ef126d-e530-4343-bb95-1262953e687b\" (UID: \"c8ef126d-e530-4343-bb95-1262953e687b\") " Nov 03 21:24:43.256850 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:43.256785 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8ef126d-e530-4343-bb95-1262953e687b-istio-data" pod "c8ef126d-e530-4343-bb95-1262953e687b" (UID: "c8ef126d-e530-4343-bb95-1262953e687b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:24:43.256850 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:43.256829 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8ef126d-e530-4343-bb95-1262953e687b-workload-socket" pod "c8ef126d-e530-4343-bb95-1262953e687b" (UID: "c8ef126d-e530-4343-bb95-1262953e687b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:24:43.257045 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:43.256991 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8ef126d-e530-4343-bb95-1262953e687b-credential-socket" pod "c8ef126d-e530-4343-bb95-1262953e687b" (UID: "c8ef126d-e530-4343-bb95-1262953e687b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:24:43.257129 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:43.257096 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8ef126d-e530-4343-bb95-1262953e687b-istiod-ca-cert" pod "c8ef126d-e530-4343-bb95-1262953e687b" (UID: "c8ef126d-e530-4343-bb95-1262953e687b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:24:43.257379 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:43.257231 4305 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ef126d-e530-4343-bb95-1262953e687b-workload-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:24:43.257379 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:43.257351 4305 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ef126d-e530-4343-bb95-1262953e687b-credential-socket\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:24:43.257379 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:43.257367 4305 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8ef126d-e530-4343-bb95-1262953e687b-workload-certs\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:24:43.257379 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:43.257381 4305 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8ef126d-e530-4343-bb95-1262953e687b-istio-data\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:24:43.258722 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:43.258695 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8ef126d-e530-4343-bb95-1262953e687b-istio-envoy" pod "c8ef126d-e530-4343-bb95-1262953e687b" (UID: "c8ef126d-e530-4343-bb95-1262953e687b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:24:43.258826 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:43.258779 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c8ef126d-e530-4343-bb95-1262953e687b-istio-podinfo" pod "c8ef126d-e530-4343-bb95-1262953e687b" (UID: "c8ef126d-e530-4343-bb95-1262953e687b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:24:43.258826 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:43.258812 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8ef126d-e530-4343-bb95-1262953e687b-istio-token" pod "c8ef126d-e530-4343-bb95-1262953e687b" (UID: "c8ef126d-e530-4343-bb95-1262953e687b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:24:43.259142 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:43.259127 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8ef126d-e530-4343-bb95-1262953e687b-kube-api-access-nbfg7" pod "c8ef126d-e530-4343-bb95-1262953e687b" (UID: "c8ef126d-e530-4343-bb95-1262953e687b"). InnerVolumeSpecName "kube-api-access-nbfg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:24:43.358531 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:43.358409 4305 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8ef126d-e530-4343-bb95-1262953e687b-istiod-ca-cert\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:24:43.358531 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:43.358483 4305 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8ef126d-e530-4343-bb95-1262953e687b-istio-envoy\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:24:43.358531 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:43.358494 4305 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8ef126d-e530-4343-bb95-1262953e687b-istio-podinfo\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:24:43.358531 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:43.358503 4305 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8ef126d-e530-4343-bb95-1262953e687b-istio-token\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:24:43.358531 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:43.358513 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbfg7\" (UniqueName: \"kubernetes.io/projected/c8ef126d-e530-4343-bb95-1262953e687b-kube-api-access-nbfg7\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:24:43.792664 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:43.792620 4305 generic.go:356] "Generic (PLEG): container finished" podID="c8ef126d-e530-4343-bb95-1262953e687b" containerID="cc44208ffaaf91b25786e283c69a6c6af6341c5bdc4445081f45251ec013a59e" exitCode=0 Nov 03 21:24:43.792881 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:43.792680 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q" Nov 03 21:24:43.792881 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:43.792670 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q" event={"ID":"c8ef126d-e530-4343-bb95-1262953e687b","Type":"ContainerDied","Data":"cc44208ffaaf91b25786e283c69a6c6af6341c5bdc4445081f45251ec013a59e"} Nov 03 21:24:43.792881 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:43.792809 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q" event={"ID":"c8ef126d-e530-4343-bb95-1262953e687b","Type":"ContainerDied","Data":"88e0c3a674123af331b759a0f4b44ea4f9f4b7f4f9e1fd80b8ba337b929ec9c8"} Nov 03 21:24:43.792881 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:43.792844 4305 scope.go:122] "RemoveContainer" containerID="cc44208ffaaf91b25786e283c69a6c6af6341c5bdc4445081f45251ec013a59e" Nov 03 21:24:43.794942 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:43.794921 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wxc9q" event={"ID":"b7172d4b-d0b8-4090-a957-07521eb40205","Type":"ContainerDied","Data":"5900b887125941cf2946605f9d3b3e121ec2f85208561227ea8945247b391f11"} Nov 03 21:24:43.795127 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:43.794957 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5900b887125941cf2946605f9d3b3e121ec2f85208561227ea8945247b391f11" Nov 03 21:24:43.795127 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:43.795028 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wxc9q" Nov 03 21:24:43.809086 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:43.809065 4305 scope.go:122] "RemoveContainer" containerID="fcb9c168f1f2da9713c0daa0fbf4824136e0db88e612006fcf7f3d5663a8fe1a" Nov 03 21:24:43.818226 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:43.818202 4305 scope.go:122] "RemoveContainer" containerID="cc44208ffaaf91b25786e283c69a6c6af6341c5bdc4445081f45251ec013a59e" Nov 03 21:24:43.818522 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:24:43.818503 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc44208ffaaf91b25786e283c69a6c6af6341c5bdc4445081f45251ec013a59e\": container with ID starting with cc44208ffaaf91b25786e283c69a6c6af6341c5bdc4445081f45251ec013a59e not found: ID does not exist" containerID="cc44208ffaaf91b25786e283c69a6c6af6341c5bdc4445081f45251ec013a59e" Nov 03 21:24:43.818583 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:43.818532 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc44208ffaaf91b25786e283c69a6c6af6341c5bdc4445081f45251ec013a59e"} err="failed to get container status \"cc44208ffaaf91b25786e283c69a6c6af6341c5bdc4445081f45251ec013a59e\": rpc error: code = NotFound desc = could not find container \"cc44208ffaaf91b25786e283c69a6c6af6341c5bdc4445081f45251ec013a59e\": container with ID starting with cc44208ffaaf91b25786e283c69a6c6af6341c5bdc4445081f45251ec013a59e not found: ID does not exist" Nov 03 21:24:43.818583 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:43.818553 4305 scope.go:122] "RemoveContainer" containerID="fcb9c168f1f2da9713c0daa0fbf4824136e0db88e612006fcf7f3d5663a8fe1a" Nov 03 21:24:43.818807 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:24:43.818790 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcb9c168f1f2da9713c0daa0fbf4824136e0db88e612006fcf7f3d5663a8fe1a\": container with ID starting with fcb9c168f1f2da9713c0daa0fbf4824136e0db88e612006fcf7f3d5663a8fe1a not found: ID does not exist" containerID="fcb9c168f1f2da9713c0daa0fbf4824136e0db88e612006fcf7f3d5663a8fe1a" Nov 03 21:24:43.818850 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:43.818815 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcb9c168f1f2da9713c0daa0fbf4824136e0db88e612006fcf7f3d5663a8fe1a"} err="failed to get container status \"fcb9c168f1f2da9713c0daa0fbf4824136e0db88e612006fcf7f3d5663a8fe1a\": rpc error: code = NotFound desc = could not find container \"fcb9c168f1f2da9713c0daa0fbf4824136e0db88e612006fcf7f3d5663a8fe1a\": container with ID starting with fcb9c168f1f2da9713c0daa0fbf4824136e0db88e612006fcf7f3d5663a8fe1a not found: ID does not exist" Nov 03 21:24:43.957586 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:43.957553 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q"] Nov 03 21:24:44.007363 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:44.007324 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-96c0ab-openshift-default-796564d969j7p8q"] Nov 03 21:24:44.894273 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:44.894240 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8ef126d-e530-4343-bb95-1262953e687b" path="/var/lib/kubelet/pods/c8ef126d-e530-4343-bb95-1262953e687b/volumes" Nov 03 21:24:46.321517 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:24:46.321480 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:24:50.209058 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:50.208990 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-96c0ab-748d4f7df2n7fk" Nov 03 21:24:50.334227 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:50.334193 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/165d07de-c15a-413b-83f1-fa25e7b899a1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/165d07de-c15a-413b-83f1-fa25e7b899a1-plugins-config-volume\") pod \"165d07de-c15a-413b-83f1-fa25e7b899a1\" (UID: \"165d07de-c15a-413b-83f1-fa25e7b899a1\") " Nov 03 21:24:50.334421 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:50.334263 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/165d07de-c15a-413b-83f1-fa25e7b899a1-kube-api-access-mt9rq\" (UniqueName: \"kubernetes.io/projected/165d07de-c15a-413b-83f1-fa25e7b899a1-kube-api-access-mt9rq\") pod \"165d07de-c15a-413b-83f1-fa25e7b899a1\" (UID: \"165d07de-c15a-413b-83f1-fa25e7b899a1\") " Nov 03 21:24:50.334696 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:50.334667 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/165d07de-c15a-413b-83f1-fa25e7b899a1-plugins-config-volume" pod "165d07de-c15a-413b-83f1-fa25e7b899a1" (UID: "165d07de-c15a-413b-83f1-fa25e7b899a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:24:50.336616 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:50.336587 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/165d07de-c15a-413b-83f1-fa25e7b899a1-kube-api-access-mt9rq" pod "165d07de-c15a-413b-83f1-fa25e7b899a1" (UID: "165d07de-c15a-413b-83f1-fa25e7b899a1"). InnerVolumeSpecName "kube-api-access-mt9rq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:24:50.435054 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:50.435002 4305 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/165d07de-c15a-413b-83f1-fa25e7b899a1-plugins-config-volume\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:24:50.435054 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:50.435048 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mt9rq\" (UniqueName: \"kubernetes.io/projected/165d07de-c15a-413b-83f1-fa25e7b899a1-kube-api-access-mt9rq\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:24:50.833666 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:50.833625 4305 generic.go:356] "Generic (PLEG): container finished" podID="165d07de-c15a-413b-83f1-fa25e7b899a1" containerID="0d19faeb1e1e154ceceff700d34c7bc5bd495ef6cbd3de30d87cd25ed9a1c4b3" exitCode=0 Nov 03 21:24:50.833852 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:50.833675 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-96c0ab-748d4f7df2n7fk" Nov 03 21:24:50.833852 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:50.833683 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-96c0ab-748d4f7df2n7fk" event={"ID":"165d07de-c15a-413b-83f1-fa25e7b899a1","Type":"ContainerDied","Data":"0d19faeb1e1e154ceceff700d34c7bc5bd495ef6cbd3de30d87cd25ed9a1c4b3"} Nov 03 21:24:50.833852 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:50.833747 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-96c0ab-748d4f7df2n7fk" event={"ID":"165d07de-c15a-413b-83f1-fa25e7b899a1","Type":"ContainerDied","Data":"8e32b2b5394f17dff0b25c369d4f170c18152a0eb47c44f798ef43c5005a8573"} Nov 03 21:24:50.833852 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:50.833779 4305 scope.go:122] "RemoveContainer" containerID="0d19faeb1e1e154ceceff700d34c7bc5bd495ef6cbd3de30d87cd25ed9a1c4b3" Nov 03 21:24:50.843572 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:50.843542 4305 scope.go:122] "RemoveContainer" containerID="0d19faeb1e1e154ceceff700d34c7bc5bd495ef6cbd3de30d87cd25ed9a1c4b3" Nov 03 21:24:50.843860 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:24:50.843844 4305 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d19faeb1e1e154ceceff700d34c7bc5bd495ef6cbd3de30d87cd25ed9a1c4b3\": container with ID starting with 0d19faeb1e1e154ceceff700d34c7bc5bd495ef6cbd3de30d87cd25ed9a1c4b3 not found: ID does not exist" containerID="0d19faeb1e1e154ceceff700d34c7bc5bd495ef6cbd3de30d87cd25ed9a1c4b3" Nov 03 21:24:50.843911 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:50.843868 4305 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d19faeb1e1e154ceceff700d34c7bc5bd495ef6cbd3de30d87cd25ed9a1c4b3"} err="failed to get container status \"0d19faeb1e1e154ceceff700d34c7bc5bd495ef6cbd3de30d87cd25ed9a1c4b3\": rpc error: code = NotFound desc = could not find container \"0d19faeb1e1e154ceceff700d34c7bc5bd495ef6cbd3de30d87cd25ed9a1c4b3\": container with ID starting with 0d19faeb1e1e154ceceff700d34c7bc5bd495ef6cbd3de30d87cd25ed9a1c4b3 not found: ID does not exist" Nov 03 21:24:50.997793 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:50.997755 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-96c0ab-748d4f7df2n7fk"] Nov 03 21:24:51.047076 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:51.047040 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-96c0ab-748d4f7df2n7fk"] Nov 03 21:24:52.893450 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:52.893398 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="165d07de-c15a-413b-83f1-fa25e7b899a1" path="/var/lib/kubelet/pods/165d07de-c15a-413b-83f1-fa25e7b899a1/volumes" Nov 03 21:24:54.184098 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:54.184061 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:24:59.048188 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:59.048154 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-czgbn"] Nov 03 21:24:59.097709 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:24:59.097670 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-czgbn"] Nov 03 21:25:00.893901 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:25:00.893867 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="424f721f-ea02-4659-bfa6-7f11bb3a901d" path="/var/lib/kubelet/pods/424f721f-ea02-4659-bfa6-7f11bb3a901d/volumes" Nov 03 21:25:04.984242 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:25:04.984203 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:25:16.321363 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:25:16.321321 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:25:32.922627 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:25:32.922596 4305 scope.go:122] "RemoveContainer" containerID="7afc971a7a4f25587ce7355d49e179f35aa85798431569a816987220b0ece5c9" Nov 03 21:25:32.931761 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:25:32.931741 4305 scope.go:122] "RemoveContainer" containerID="302786a7b11fe2208c90284f84a79fb881bd52fa24194fde3ada952877629117" Nov 03 21:25:46.321569 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:25:46.321534 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:26:16.323160 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:26:16.323122 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:26:17.486419 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:26:17.486380 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zffsq"] Nov 03 21:26:17.487088 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:26:17.487068 4305 cpu_manager.go:414] "containerMap: removing container" podUID="c8ef126d-e530-4343-bb95-1262953e687b" containerName="istio-proxy" Nov 03 21:26:17.487159 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:26:17.487093 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8ef126d-e530-4343-bb95-1262953e687b" containerName="istio-proxy" Nov 03 21:26:17.487159 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:26:17.487121 4305 cpu_manager.go:414] "containerMap: removing container" podUID="b7172d4b-d0b8-4090-a957-07521eb40205" containerName="revoke-keys" Nov 03 21:26:17.487159 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:26:17.487126 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7172d4b-d0b8-4090-a957-07521eb40205" containerName="revoke-keys" Nov 03 21:26:17.487159 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:26:17.487142 4305 cpu_manager.go:414] "containerMap: removing container" podUID="165d07de-c15a-413b-83f1-fa25e7b899a1" containerName="payload-processing" Nov 03 21:26:17.487159 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:26:17.487146 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="165d07de-c15a-413b-83f1-fa25e7b899a1" containerName="payload-processing" Nov 03 21:26:17.487322 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:26:17.487211 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="165d07de-c15a-413b-83f1-fa25e7b899a1" containerName="payload-processing" Nov 03 21:26:17.487322 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:26:17.487219 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8ef126d-e530-4343-bb95-1262953e687b" containerName="istio-proxy" Nov 03 21:26:17.487322 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:26:17.487232 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7172d4b-d0b8-4090-a957-07521eb40205" containerName="revoke-keys" Nov 03 21:26:17.490623 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:26:17.490607 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zffsq" Nov 03 21:26:17.544214 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:26:17.544180 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-6p7qs\"" Nov 03 21:26:17.578548 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:26:17.578506 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2cjc\" (UniqueName: \"kubernetes.io/projected/c7af8a41-29dc-40bd-9065-074de2b99455-kube-api-access-b2cjc\") pod \"kuadrant-operator-controller-manager-678d87766f-zffsq\" (UID: \"c7af8a41-29dc-40bd-9065-074de2b99455\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zffsq" Nov 03 21:26:17.578734 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:26:17.578659 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c7af8a41-29dc-40bd-9065-074de2b99455-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-zffsq\" (UID: \"c7af8a41-29dc-40bd-9065-074de2b99455\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zffsq" Nov 03 21:26:17.594975 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:26:17.594940 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zffsq"] Nov 03 21:26:17.680014 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:26:17.679970 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2cjc\" (UniqueName: \"kubernetes.io/projected/c7af8a41-29dc-40bd-9065-074de2b99455-kube-api-access-b2cjc\") pod \"kuadrant-operator-controller-manager-678d87766f-zffsq\" (UID: \"c7af8a41-29dc-40bd-9065-074de2b99455\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zffsq" Nov 03 21:26:17.680219 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:26:17.680037 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c7af8a41-29dc-40bd-9065-074de2b99455-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-zffsq\" (UID: \"c7af8a41-29dc-40bd-9065-074de2b99455\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zffsq" Nov 03 21:26:17.680399 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:26:17.680382 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c7af8a41-29dc-40bd-9065-074de2b99455-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-zffsq\" (UID: \"c7af8a41-29dc-40bd-9065-074de2b99455\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zffsq" Nov 03 21:26:17.741163 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:26:17.741081 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2cjc\" (UniqueName: \"kubernetes.io/projected/c7af8a41-29dc-40bd-9065-074de2b99455-kube-api-access-b2cjc\") pod \"kuadrant-operator-controller-manager-678d87766f-zffsq\" (UID: \"c7af8a41-29dc-40bd-9065-074de2b99455\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zffsq" Nov 03 21:26:17.804065 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:26:17.804032 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zffsq" Nov 03 21:26:18.291901 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:26:18.291868 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zffsq"] Nov 03 21:26:18.294261 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:26:18.294227 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7af8a41_29dc_40bd_9065_074de2b99455.slice/crio-6b92e6243419f2adedce6db0523948e6162418ea8173a89266d1cd8f5b6ebb7c WatchSource:0}: Error finding container 6b92e6243419f2adedce6db0523948e6162418ea8173a89266d1cd8f5b6ebb7c: Status 404 returned error can't find the container with id 6b92e6243419f2adedce6db0523948e6162418ea8173a89266d1cd8f5b6ebb7c Nov 03 21:26:18.318863 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:26:18.318810 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zffsq" event={"ID":"c7af8a41-29dc-40bd-9065-074de2b99455","Type":"ContainerStarted","Data":"6b92e6243419f2adedce6db0523948e6162418ea8173a89266d1cd8f5b6ebb7c"} Nov 03 21:26:19.325790 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:26:19.325747 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zffsq" event={"ID":"c7af8a41-29dc-40bd-9065-074de2b99455","Type":"ContainerStarted","Data":"e9b9f36d20d79afd32fe1ee70d1c5c6dee48da91a248a4a5ce17347bbe18705c"} Nov 03 21:26:19.326252 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:26:19.325945 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zffsq" Nov 03 21:26:19.448741 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:26:19.448677 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zffsq" podStartSLOduration=2.448658886 podStartE2EDuration="2.448658886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:26:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:26:19.447185648 +0000 UTC m=+2814.672505454" watchObservedRunningTime="2026-11-03 21:26:19.448658886 +0000 UTC m=+2814.673978678" Nov 03 21:26:30.333195 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:26:30.333163 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zffsq" Nov 03 21:26:32.964758 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:26:32.964731 4305 scope.go:122] "RemoveContainer" containerID="6e59c7d74890f7a5e32add8b385488d0996c40a2b28d28519832ddeeaaed2bc8" Nov 03 21:26:32.974028 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:26:32.974007 4305 scope.go:122] "RemoveContainer" containerID="36b2ee1052cffbc322ae23d2f50263c73afa39fb21ecdda7c9e92e7b28acda04" Nov 03 21:26:34.049645 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:26:34.049612 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qthgc"] Nov 03 21:26:34.099206 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:26:34.099173 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qthgc"] Nov 03 21:26:34.894384 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:26:34.894351 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="611e686a-7db8-4a27-9798-e098067154fe" path="/var/lib/kubelet/pods/611e686a-7db8-4a27-9798-e098067154fe/volumes" Nov 03 21:26:38.893629 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:26:38.893542 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:26:45.049639 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:26:45.049600 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-l7t5m"] Nov 03 21:26:45.099080 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:26:45.099040 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-l7t5m"] Nov 03 21:26:46.321930 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:26:46.321896 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:26:46.896351 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:26:46.896320 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="779c1871-8b66-477c-b9b4-1bb706571d65" path="/var/lib/kubelet/pods/779c1871-8b66-477c-b9b4-1bb706571d65/volumes" Nov 03 21:26:56.376752 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:26:56.376706 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:27:06.367268 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:27:06.367226 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:27:16.320806 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:27:16.320767 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:27:25.402715 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:27:25.402676 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:27:32.997641 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:27:32.997605 4305 scope.go:122] "RemoveContainer" containerID="f728d4976ad4d0cd1070055db8ba188918e84c8f691a74cf33990f9d9070b372" Nov 03 21:27:33.011344 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:27:33.011274 4305 scope.go:122] "RemoveContainer" containerID="05265995defacb071668c0ef2e7821ccaad8b0c23049ee7f7dffa3f255926944" Nov 03 21:27:43.874609 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:27:43.874567 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:27:46.321804 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:27:46.321764 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:28:02.885414 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:28:02.885382 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:28:16.322065 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:28:16.322024 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:28:27.876745 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:28:27.876650 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:28:33.040467 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:28:33.040420 4305 scope.go:122] "RemoveContainer" containerID="d7ffe6678a3dbcc4790d5dfdd4706bb44e99b795ea8670f995e463e78b667597" Nov 03 21:28:33.050365 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:28:33.050345 4305 scope.go:122] "RemoveContainer" containerID="b6c0b523326e29bf721f03529223ae454b06ecea7a9d6d59086ddf8969807b44" Nov 03 21:28:46.320260 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:28:46.320217 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:28:55.798673 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:28:55.798639 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:29:05.086317 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:29:05.086273 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-189e7f0a-lqv78"] Nov 03 21:29:05.135096 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:29:05.135061 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-189e7f0a-lqv78"] Nov 03 21:29:06.895414 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:29:06.895377 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b9c18a0-ad8f-49ed-a337-2128333f6332" path="/var/lib/kubelet/pods/5b9c18a0-ad8f-49ed-a337-2128333f6332/volumes" Nov 03 21:29:16.322969 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:29:16.322931 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:29:16.572373 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:29:16.572334 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:29:33.276830 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:29:33.276785 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:29:44.045489 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:29:44.045392 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wxc9q"] Nov 03 21:29:44.094257 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:29:44.094224 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wxc9q"] Nov 03 21:29:44.894083 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:29:44.894051 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7172d4b-d0b8-4090-a957-07521eb40205" path="/var/lib/kubelet/pods/b7172d4b-d0b8-4090-a957-07521eb40205/volumes" Nov 03 21:29:46.320956 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:29:46.320913 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:30:16.320839 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:30:16.320793 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:30:18.307396 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:30:18.307355 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:30:33.082397 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:30:33.082361 4305 scope.go:122] "RemoveContainer" containerID="e331d50421df6d7796da266446a1f2383de18d263c09c3dccc380399eef75a4e" Nov 03 21:30:33.091282 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:30:33.091256 4305 scope.go:122] "RemoveContainer" containerID="029dfc716524daf222976cb611a89fc8e6e7330d2e33bc5bb8635728fb76a2fc" Nov 03 21:30:45.662913 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:30:45.662878 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:30:46.329351 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:30:46.329310 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:30:59.678342 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:30:59.678306 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:31:16.321568 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:31:16.321532 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:31:17.570032 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:31:17.569998 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:31:29.964207 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:31:29.964161 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:31:33.118049 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:31:33.117757 4305 scope.go:122] "RemoveContainer" containerID="9b528d3beb3d581c84ceab8aa8e3095327dbe4e1ccb9c9070261f6941bdd65e4" Nov 03 21:31:44.279852 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:31:44.279810 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:31:46.347588 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:31:46.347544 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:32:07.500676 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:32:07.500637 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:32:16.321860 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:32:16.321819 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:32:41.795882 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:32:41.795668 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:32:46.326662 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:32:46.326619 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:32:54.279349 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:32:54.279306 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb2b291b-l8p7q"] Nov 03 21:32:54.280031 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:32:54.280008 4305 cpu_manager.go:414] "containerMap: removing container" podUID="c8ef126d-e530-4343-bb95-1262953e687b" containerName="istio-proxy" Nov 03 21:32:54.280120 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:32:54.280034 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8ef126d-e530-4343-bb95-1262953e687b" containerName="istio-proxy" Nov 03 21:32:54.280208 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:32:54.280195 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8ef126d-e530-4343-bb95-1262953e687b" containerName="istio-proxy" Nov 03 21:32:54.283506 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:32:54.283483 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb2b291b-l8p7q" Nov 03 21:32:54.338167 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:32:54.338101 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:32:54.338167 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:32:54.338139 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:32:54.343742 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:32:54.343713 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ebce9f7-19a5-4e91-8ce6-df22199baea9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-bb2b291b-l8p7q\" (UID: \"4ebce9f7-19a5-4e91-8ce6-df22199baea9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb2b291b-l8p7q" Nov 03 21:32:54.343901 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:32:54.343868 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz5r9\" (UniqueName: \"kubernetes.io/projected/4ebce9f7-19a5-4e91-8ce6-df22199baea9-kube-api-access-cz5r9\") pod \"maas-api-revoke-keys-e2e-ait-bb2b291b-l8p7q\" (UID: \"4ebce9f7-19a5-4e91-8ce6-df22199baea9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb2b291b-l8p7q" Nov 03 21:32:54.344022 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:32:54.343992 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ebce9f7-19a5-4e91-8ce6-df22199baea9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-bb2b291b-l8p7q\" (UID: \"4ebce9f7-19a5-4e91-8ce6-df22199baea9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb2b291b-l8p7q" Nov 03 21:32:54.387795 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:32:54.387757 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:32:54.387990 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:32:54.387828 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ftgz\"" Nov 03 21:32:54.389943 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:32:54.389917 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb2b291b-l8p7q"] Nov 03 21:32:54.444915 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:32:54.444880 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ebce9f7-19a5-4e91-8ce6-df22199baea9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-bb2b291b-l8p7q\" (UID: \"4ebce9f7-19a5-4e91-8ce6-df22199baea9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb2b291b-l8p7q" Nov 03 21:32:54.445146 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:32:54.444992 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz5r9\" (UniqueName: \"kubernetes.io/projected/4ebce9f7-19a5-4e91-8ce6-df22199baea9-kube-api-access-cz5r9\") pod \"maas-api-revoke-keys-e2e-ait-bb2b291b-l8p7q\" (UID: \"4ebce9f7-19a5-4e91-8ce6-df22199baea9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb2b291b-l8p7q" Nov 03 21:32:54.445146 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:32:54.445059 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ebce9f7-19a5-4e91-8ce6-df22199baea9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-bb2b291b-l8p7q\" (UID: \"4ebce9f7-19a5-4e91-8ce6-df22199baea9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb2b291b-l8p7q" Nov 03 21:32:54.445812 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:32:54.445741 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ebce9f7-19a5-4e91-8ce6-df22199baea9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-bb2b291b-l8p7q\" (UID: \"4ebce9f7-19a5-4e91-8ce6-df22199baea9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb2b291b-l8p7q" Nov 03 21:32:54.447513 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:32:54.447488 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ebce9f7-19a5-4e91-8ce6-df22199baea9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-bb2b291b-l8p7q\" (UID: \"4ebce9f7-19a5-4e91-8ce6-df22199baea9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb2b291b-l8p7q" Nov 03 21:32:54.505724 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:32:54.505691 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz5r9\" (UniqueName: \"kubernetes.io/projected/4ebce9f7-19a5-4e91-8ce6-df22199baea9-kube-api-access-cz5r9\") pod \"maas-api-revoke-keys-e2e-ait-bb2b291b-l8p7q\" (UID: \"4ebce9f7-19a5-4e91-8ce6-df22199baea9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb2b291b-l8p7q" Nov 03 21:32:54.596626 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:32:54.596542 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb2b291b-l8p7q" Nov 03 21:32:54.974210 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:32:54.974098 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb2b291b-l8p7q"] Nov 03 21:32:54.976706 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:32:54.976677 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ebce9f7_19a5_4e91_8ce6_df22199baea9.slice/crio-5d918b26192fe98a53dcda10770b5feb863723cac16c784c851eb87fecb42de7 WatchSource:0}: Error finding container 5d918b26192fe98a53dcda10770b5feb863723cac16c784c851eb87fecb42de7: Status 404 returned error can't find the container with id 5d918b26192fe98a53dcda10770b5feb863723cac16c784c851eb87fecb42de7 Nov 03 21:32:55.565213 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:32:55.565125 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb2b291b-l8p7q" event={"ID":"4ebce9f7-19a5-4e91-8ce6-df22199baea9","Type":"ContainerStarted","Data":"8ef6135fd9cf91c26f9982168649ba271acccad15d3e3236ab3c377e6a6b2dfc"} Nov 03 21:32:55.565213 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:32:55.565193 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb2b291b-l8p7q" event={"ID":"4ebce9f7-19a5-4e91-8ce6-df22199baea9","Type":"ContainerStarted","Data":"5d918b26192fe98a53dcda10770b5feb863723cac16c784c851eb87fecb42de7"} Nov 03 21:32:55.681617 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:32:55.681557 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb2b291b-l8p7q" podStartSLOduration=1.681544457 podStartE2EDuration="1.681544457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:32:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:32:55.680266081 +0000 UTC m=+3210.905585872" watchObservedRunningTime="2026-11-03 21:32:55.681544457 +0000 UTC m=+3210.906864247" Nov 03 21:32:56.572310 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:32:56.572269 4305 generic.go:356] "Generic (PLEG): container finished" podID="4ebce9f7-19a5-4e91-8ce6-df22199baea9" containerID="8ef6135fd9cf91c26f9982168649ba271acccad15d3e3236ab3c377e6a6b2dfc" exitCode=0 Nov 03 21:32:56.572805 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:32:56.572367 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb2b291b-l8p7q" event={"ID":"4ebce9f7-19a5-4e91-8ce6-df22199baea9","Type":"ContainerDied","Data":"8ef6135fd9cf91c26f9982168649ba271acccad15d3e3236ab3c377e6a6b2dfc"} Nov 03 21:32:57.726940 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:32:57.726916 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb2b291b-l8p7q" Nov 03 21:32:57.779131 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:32:57.779093 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ebce9f7-19a5-4e91-8ce6-df22199baea9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ebce9f7-19a5-4e91-8ce6-df22199baea9-maas-api-xks-ca\") pod \"4ebce9f7-19a5-4e91-8ce6-df22199baea9\" (UID: \"4ebce9f7-19a5-4e91-8ce6-df22199baea9\") " Nov 03 21:32:57.779344 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:32:57.779205 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ebce9f7-19a5-4e91-8ce6-df22199baea9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ebce9f7-19a5-4e91-8ce6-df22199baea9-maas-api-openshift-ca\") pod \"4ebce9f7-19a5-4e91-8ce6-df22199baea9\" (UID: \"4ebce9f7-19a5-4e91-8ce6-df22199baea9\") " Nov 03 21:32:57.779344 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:32:57.779238 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ebce9f7-19a5-4e91-8ce6-df22199baea9-kube-api-access-cz5r9\" (UniqueName: \"kubernetes.io/projected/4ebce9f7-19a5-4e91-8ce6-df22199baea9-kube-api-access-cz5r9\") pod \"4ebce9f7-19a5-4e91-8ce6-df22199baea9\" (UID: \"4ebce9f7-19a5-4e91-8ce6-df22199baea9\") " Nov 03 21:32:57.779673 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:32:57.779649 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ebce9f7-19a5-4e91-8ce6-df22199baea9-maas-api-openshift-ca" pod "4ebce9f7-19a5-4e91-8ce6-df22199baea9" (UID: "4ebce9f7-19a5-4e91-8ce6-df22199baea9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:32:57.781620 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:32:57.781595 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ebce9f7-19a5-4e91-8ce6-df22199baea9-maas-api-xks-ca" pod "4ebce9f7-19a5-4e91-8ce6-df22199baea9" (UID: "4ebce9f7-19a5-4e91-8ce6-df22199baea9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:32:57.781710 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:32:57.781651 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ebce9f7-19a5-4e91-8ce6-df22199baea9-kube-api-access-cz5r9" pod "4ebce9f7-19a5-4e91-8ce6-df22199baea9" (UID: "4ebce9f7-19a5-4e91-8ce6-df22199baea9"). InnerVolumeSpecName "kube-api-access-cz5r9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:32:57.880179 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:32:57.880095 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ebce9f7-19a5-4e91-8ce6-df22199baea9-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:32:57.880179 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:32:57.880130 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ebce9f7-19a5-4e91-8ce6-df22199baea9-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:32:57.880179 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:32:57.880144 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cz5r9\" (UniqueName: \"kubernetes.io/projected/4ebce9f7-19a5-4e91-8ce6-df22199baea9-kube-api-access-cz5r9\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:32:58.584607 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:32:58.584572 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb2b291b-l8p7q" Nov 03 21:32:58.584607 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:32:58.584606 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb2b291b-l8p7q" event={"ID":"4ebce9f7-19a5-4e91-8ce6-df22199baea9","Type":"ContainerDied","Data":"5d918b26192fe98a53dcda10770b5feb863723cac16c784c851eb87fecb42de7"} Nov 03 21:32:58.584886 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:32:58.584655 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d918b26192fe98a53dcda10770b5feb863723cac16c784c851eb87fecb42de7" Nov 03 21:33:14.276779 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:33:14.276744 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb2b291b-l8p7q"] Nov 03 21:33:14.326015 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:33:14.325985 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb2b291b-l8p7q"] Nov 03 21:33:14.894535 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:33:14.894500 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ebce9f7-19a5-4e91-8ce6-df22199baea9" path="/var/lib/kubelet/pods/4ebce9f7-19a5-4e91-8ce6-df22199baea9/volumes" Nov 03 21:33:16.321286 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:33:16.321248 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:33:18.956349 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:33:18.956311 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:33:29.619754 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:33:29.619719 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:33:46.323285 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:33:46.323246 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:34:16.321844 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:34:16.321809 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:34:46.321119 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:34:46.321078 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:35:03.939838 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:35:03.939794 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-220932-d2kt6"] Nov 03 21:35:03.940314 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:35:03.940301 4305 cpu_manager.go:414] "containerMap: removing container" podUID="4ebce9f7-19a5-4e91-8ce6-df22199baea9" containerName="revoke-keys" Nov 03 21:35:03.940314 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:35:03.940314 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ebce9f7-19a5-4e91-8ce6-df22199baea9" containerName="revoke-keys" Nov 03 21:35:03.940406 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:35:03.940396 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ebce9f7-19a5-4e91-8ce6-df22199baea9" containerName="revoke-keys" Nov 03 21:35:03.943572 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:35:03.943556 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-220932-d2kt6" Nov 03 21:35:03.999367 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:35:03.999338 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:35:03.999367 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:35:03.999335 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:35:03.999367 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:35:03.999377 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:35:04.049153 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:35:04.049124 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ftgz\"" Nov 03 21:35:04.051467 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:35:04.051415 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-220932-d2kt6"] Nov 03 21:35:04.117295 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:35:04.117260 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/347ab5cb-c006-4b14-8334-54fd40f19b0f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-220932-d2kt6\" (UID: \"347ab5cb-c006-4b14-8334-54fd40f19b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-220932-d2kt6" Nov 03 21:35:04.117477 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:35:04.117372 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/347ab5cb-c006-4b14-8334-54fd40f19b0f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-220932-d2kt6\" (UID: \"347ab5cb-c006-4b14-8334-54fd40f19b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-220932-d2kt6" Nov 03 21:35:04.117477 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:35:04.117418 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxcvd\" (UniqueName: \"kubernetes.io/projected/347ab5cb-c006-4b14-8334-54fd40f19b0f-kube-api-access-zxcvd\") pod \"maas-api-revoke-keys-e2e-authcfg-http-220932-d2kt6\" (UID: \"347ab5cb-c006-4b14-8334-54fd40f19b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-220932-d2kt6" Nov 03 21:35:04.218167 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:35:04.218076 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/347ab5cb-c006-4b14-8334-54fd40f19b0f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-220932-d2kt6\" (UID: \"347ab5cb-c006-4b14-8334-54fd40f19b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-220932-d2kt6" Nov 03 21:35:04.218167 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:35:04.218138 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/347ab5cb-c006-4b14-8334-54fd40f19b0f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-220932-d2kt6\" (UID: \"347ab5cb-c006-4b14-8334-54fd40f19b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-220932-d2kt6" Nov 03 21:35:04.218387 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:35:04.218179 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxcvd\" (UniqueName: \"kubernetes.io/projected/347ab5cb-c006-4b14-8334-54fd40f19b0f-kube-api-access-zxcvd\") pod \"maas-api-revoke-keys-e2e-authcfg-http-220932-d2kt6\" (UID: \"347ab5cb-c006-4b14-8334-54fd40f19b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-220932-d2kt6" Nov 03 21:35:04.218783 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:35:04.218757 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/347ab5cb-c006-4b14-8334-54fd40f19b0f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-220932-d2kt6\" (UID: \"347ab5cb-c006-4b14-8334-54fd40f19b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-220932-d2kt6" Nov 03 21:35:04.220686 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:35:04.220661 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/347ab5cb-c006-4b14-8334-54fd40f19b0f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-220932-d2kt6\" (UID: \"347ab5cb-c006-4b14-8334-54fd40f19b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-220932-d2kt6" Nov 03 21:35:04.279120 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:35:04.279090 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxcvd\" (UniqueName: \"kubernetes.io/projected/347ab5cb-c006-4b14-8334-54fd40f19b0f-kube-api-access-zxcvd\") pod \"maas-api-revoke-keys-e2e-authcfg-http-220932-d2kt6\" (UID: \"347ab5cb-c006-4b14-8334-54fd40f19b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-220932-d2kt6" Nov 03 21:35:04.556811 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:35:04.556732 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-220932-d2kt6" Nov 03 21:35:04.931034 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:35:04.931000 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-220932-d2kt6"] Nov 03 21:35:04.933863 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:35:04.933838 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod347ab5cb_c006_4b14_8334_54fd40f19b0f.slice/crio-3638b204f6b8c4ec9deb3eb2ca290ea232cc81216a2cd3cedb0018cacdd983cd WatchSource:0}: Error finding container 3638b204f6b8c4ec9deb3eb2ca290ea232cc81216a2cd3cedb0018cacdd983cd: Status 404 returned error can't find the container with id 3638b204f6b8c4ec9deb3eb2ca290ea232cc81216a2cd3cedb0018cacdd983cd Nov 03 21:35:05.206311 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:35:05.206222 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-220932-d2kt6" event={"ID":"347ab5cb-c006-4b14-8334-54fd40f19b0f","Type":"ContainerStarted","Data":"3638b204f6b8c4ec9deb3eb2ca290ea232cc81216a2cd3cedb0018cacdd983cd"} Nov 03 21:35:06.212491 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:35:06.212425 4305 generic.go:356] "Generic (PLEG): container finished" podID="347ab5cb-c006-4b14-8334-54fd40f19b0f" containerID="1710bb38adcd931e3dd70fb3865e231454efa9e3b5e051ac01f55c1c402cefa3" exitCode=0 Nov 03 21:35:06.212491 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:35:06.212474 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-220932-d2kt6" event={"ID":"347ab5cb-c006-4b14-8334-54fd40f19b0f","Type":"ContainerDied","Data":"1710bb38adcd931e3dd70fb3865e231454efa9e3b5e051ac01f55c1c402cefa3"} Nov 03 21:35:07.364722 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:35:07.364698 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-220932-d2kt6" Nov 03 21:35:07.444132 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:35:07.444104 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/347ab5cb-c006-4b14-8334-54fd40f19b0f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/347ab5cb-c006-4b14-8334-54fd40f19b0f-maas-api-openshift-ca\") pod \"347ab5cb-c006-4b14-8334-54fd40f19b0f\" (UID: \"347ab5cb-c006-4b14-8334-54fd40f19b0f\") " Nov 03 21:35:07.444357 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:35:07.444163 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/347ab5cb-c006-4b14-8334-54fd40f19b0f-kube-api-access-zxcvd\" (UniqueName: \"kubernetes.io/projected/347ab5cb-c006-4b14-8334-54fd40f19b0f-kube-api-access-zxcvd\") pod \"347ab5cb-c006-4b14-8334-54fd40f19b0f\" (UID: \"347ab5cb-c006-4b14-8334-54fd40f19b0f\") " Nov 03 21:35:07.444357 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:35:07.444209 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/347ab5cb-c006-4b14-8334-54fd40f19b0f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/347ab5cb-c006-4b14-8334-54fd40f19b0f-maas-api-xks-ca\") pod \"347ab5cb-c006-4b14-8334-54fd40f19b0f\" (UID: \"347ab5cb-c006-4b14-8334-54fd40f19b0f\") " Nov 03 21:35:07.444567 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:35:07.444542 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/347ab5cb-c006-4b14-8334-54fd40f19b0f-maas-api-openshift-ca" pod "347ab5cb-c006-4b14-8334-54fd40f19b0f" (UID: "347ab5cb-c006-4b14-8334-54fd40f19b0f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:35:07.446397 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:35:07.446372 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/347ab5cb-c006-4b14-8334-54fd40f19b0f-maas-api-xks-ca" pod "347ab5cb-c006-4b14-8334-54fd40f19b0f" (UID: "347ab5cb-c006-4b14-8334-54fd40f19b0f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:35:07.446516 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:35:07.446467 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/347ab5cb-c006-4b14-8334-54fd40f19b0f-kube-api-access-zxcvd" pod "347ab5cb-c006-4b14-8334-54fd40f19b0f" (UID: "347ab5cb-c006-4b14-8334-54fd40f19b0f"). InnerVolumeSpecName "kube-api-access-zxcvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:35:07.545129 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:35:07.545034 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/347ab5cb-c006-4b14-8334-54fd40f19b0f-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:35:07.545129 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:35:07.545079 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/347ab5cb-c006-4b14-8334-54fd40f19b0f-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:35:07.545129 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:35:07.545090 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxcvd\" (UniqueName: \"kubernetes.io/projected/347ab5cb-c006-4b14-8334-54fd40f19b0f-kube-api-access-zxcvd\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:35:08.226578 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:35:08.226550 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-220932-d2kt6" Nov 03 21:35:08.226763 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:35:08.226587 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-220932-d2kt6" event={"ID":"347ab5cb-c006-4b14-8334-54fd40f19b0f","Type":"ContainerDied","Data":"3638b204f6b8c4ec9deb3eb2ca290ea232cc81216a2cd3cedb0018cacdd983cd"} Nov 03 21:35:08.226763 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:35:08.226633 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3638b204f6b8c4ec9deb3eb2ca290ea232cc81216a2cd3cedb0018cacdd983cd" Nov 03 21:35:16.320387 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:35:16.320341 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:35:26.251428 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:35:26.251392 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-220932-d2kt6"] Nov 03 21:35:26.300907 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:35:26.300869 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-220932-d2kt6"] Nov 03 21:35:26.894197 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:35:26.894165 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="347ab5cb-c006-4b14-8334-54fd40f19b0f" path="/var/lib/kubelet/pods/347ab5cb-c006-4b14-8334-54fd40f19b0f/volumes" Nov 03 21:35:29.774106 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:35:29.774056 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:35:38.282781 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:35:38.282695 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:35:46.320724 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:35:46.320679 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:35:51.973027 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:35:51.972989 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:35:59.892094 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:35:59.892047 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7vnw9"] Nov 03 21:35:59.892651 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:35:59.892496 4305 cpu_manager.go:414] "containerMap: removing container" podUID="347ab5cb-c006-4b14-8334-54fd40f19b0f" containerName="revoke-keys" Nov 03 21:35:59.892651 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:35:59.892508 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="347ab5cb-c006-4b14-8334-54fd40f19b0f" containerName="revoke-keys" Nov 03 21:35:59.892651 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:35:59.892585 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="347ab5cb-c006-4b14-8334-54fd40f19b0f" containerName="revoke-keys" Nov 03 21:35:59.895736 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:35:59.895717 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7vnw9" Nov 03 21:35:59.999977 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:35:59.999917 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:35:59.999977 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:35:59.999964 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:35:59.999977 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:35:59.999922 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ftgz\"" Nov 03 21:36:00.000299 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:00.000200 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:36:00.002187 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:00.002157 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7vnw9"] Nov 03 21:36:00.074765 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:00.074725 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bedb66f-11ea-42e1-a10c-3640bc51828b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-7vnw9\" (UID: \"7bedb66f-11ea-42e1-a10c-3640bc51828b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7vnw9" Nov 03 21:36:00.074765 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:00.074766 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j2pk\" (UniqueName: \"kubernetes.io/projected/7bedb66f-11ea-42e1-a10c-3640bc51828b-kube-api-access-4j2pk\") pod \"maas-api-revoke-sub-f03629db-7vnw9\" (UID: \"7bedb66f-11ea-42e1-a10c-3640bc51828b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7vnw9" Nov 03 21:36:00.075026 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:00.074933 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bedb66f-11ea-42e1-a10c-3640bc51828b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-7vnw9\" (UID: \"7bedb66f-11ea-42e1-a10c-3640bc51828b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7vnw9" Nov 03 21:36:00.176184 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:00.176090 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bedb66f-11ea-42e1-a10c-3640bc51828b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-7vnw9\" (UID: \"7bedb66f-11ea-42e1-a10c-3640bc51828b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7vnw9" Nov 03 21:36:00.176184 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:00.176136 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j2pk\" (UniqueName: \"kubernetes.io/projected/7bedb66f-11ea-42e1-a10c-3640bc51828b-kube-api-access-4j2pk\") pod \"maas-api-revoke-sub-f03629db-7vnw9\" (UID: \"7bedb66f-11ea-42e1-a10c-3640bc51828b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7vnw9" Nov 03 21:36:00.176184 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:00.176181 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bedb66f-11ea-42e1-a10c-3640bc51828b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-7vnw9\" (UID: \"7bedb66f-11ea-42e1-a10c-3640bc51828b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7vnw9" Nov 03 21:36:00.176912 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:00.176883 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bedb66f-11ea-42e1-a10c-3640bc51828b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-7vnw9\" (UID: \"7bedb66f-11ea-42e1-a10c-3640bc51828b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7vnw9" Nov 03 21:36:00.178668 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:00.178645 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bedb66f-11ea-42e1-a10c-3640bc51828b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-7vnw9\" (UID: \"7bedb66f-11ea-42e1-a10c-3640bc51828b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7vnw9" Nov 03 21:36:00.236851 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:00.236817 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j2pk\" (UniqueName: \"kubernetes.io/projected/7bedb66f-11ea-42e1-a10c-3640bc51828b-kube-api-access-4j2pk\") pod \"maas-api-revoke-sub-f03629db-7vnw9\" (UID: \"7bedb66f-11ea-42e1-a10c-3640bc51828b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7vnw9" Nov 03 21:36:00.509484 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:00.509361 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7vnw9" Nov 03 21:36:00.886064 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:00.886034 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7vnw9"] Nov 03 21:36:00.888834 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:36:00.888807 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bedb66f_11ea_42e1_a10c_3640bc51828b.slice/crio-7476c2bdda52f4fffa8812858bc73ad143362c081d1f5e73b48e9ffd4e85c3a8 WatchSource:0}: Error finding container 7476c2bdda52f4fffa8812858bc73ad143362c081d1f5e73b48e9ffd4e85c3a8: Status 404 returned error can't find the container with id 7476c2bdda52f4fffa8812858bc73ad143362c081d1f5e73b48e9ffd4e85c3a8 Nov 03 21:36:01.522920 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:01.522827 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7vnw9" event={"ID":"7bedb66f-11ea-42e1-a10c-3640bc51828b","Type":"ContainerStarted","Data":"42b4a169e1b3a10fc2b490515ca9e4cb15e39b48434d5e48dc2372be09104494"} Nov 03 21:36:01.523296 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:01.522931 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7vnw9" event={"ID":"7bedb66f-11ea-42e1-a10c-3640bc51828b","Type":"ContainerStarted","Data":"7476c2bdda52f4fffa8812858bc73ad143362c081d1f5e73b48e9ffd4e85c3a8"} Nov 03 21:36:01.640146 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:01.640085 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7vnw9" podStartSLOduration=2.640070188 podStartE2EDuration="2.640070188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:35:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:36:01.637820706 +0000 UTC m=+3396.863140496" watchObservedRunningTime="2026-11-03 21:36:01.640070188 +0000 UTC m=+3396.865389978" Nov 03 21:36:02.530278 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:02.530238 4305 generic.go:356] "Generic (PLEG): container finished" podID="7bedb66f-11ea-42e1-a10c-3640bc51828b" containerID="42b4a169e1b3a10fc2b490515ca9e4cb15e39b48434d5e48dc2372be09104494" exitCode=0 Nov 03 21:36:02.530723 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:02.530321 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7vnw9" event={"ID":"7bedb66f-11ea-42e1-a10c-3640bc51828b","Type":"ContainerDied","Data":"42b4a169e1b3a10fc2b490515ca9e4cb15e39b48434d5e48dc2372be09104494"} Nov 03 21:36:03.683513 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:03.683490 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7vnw9" Nov 03 21:36:03.711162 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:03.711132 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bedb66f-11ea-42e1-a10c-3640bc51828b-kube-api-access-4j2pk\" (UniqueName: \"kubernetes.io/projected/7bedb66f-11ea-42e1-a10c-3640bc51828b-kube-api-access-4j2pk\") pod \"7bedb66f-11ea-42e1-a10c-3640bc51828b\" (UID: \"7bedb66f-11ea-42e1-a10c-3640bc51828b\") " Nov 03 21:36:03.711341 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:03.711284 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bedb66f-11ea-42e1-a10c-3640bc51828b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bedb66f-11ea-42e1-a10c-3640bc51828b-maas-api-xks-ca\") pod \"7bedb66f-11ea-42e1-a10c-3640bc51828b\" (UID: \"7bedb66f-11ea-42e1-a10c-3640bc51828b\") " Nov 03 21:36:03.711341 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:03.711316 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bedb66f-11ea-42e1-a10c-3640bc51828b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bedb66f-11ea-42e1-a10c-3640bc51828b-maas-api-openshift-ca\") pod \"7bedb66f-11ea-42e1-a10c-3640bc51828b\" (UID: \"7bedb66f-11ea-42e1-a10c-3640bc51828b\") " Nov 03 21:36:03.711728 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:03.711702 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bedb66f-11ea-42e1-a10c-3640bc51828b-maas-api-openshift-ca" pod "7bedb66f-11ea-42e1-a10c-3640bc51828b" (UID: "7bedb66f-11ea-42e1-a10c-3640bc51828b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:36:03.713526 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:03.713500 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bedb66f-11ea-42e1-a10c-3640bc51828b-kube-api-access-4j2pk" pod "7bedb66f-11ea-42e1-a10c-3640bc51828b" (UID: "7bedb66f-11ea-42e1-a10c-3640bc51828b"). InnerVolumeSpecName "kube-api-access-4j2pk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:36:03.713643 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:03.713501 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bedb66f-11ea-42e1-a10c-3640bc51828b-maas-api-xks-ca" pod "7bedb66f-11ea-42e1-a10c-3640bc51828b" (UID: "7bedb66f-11ea-42e1-a10c-3640bc51828b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:36:03.812825 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:03.812741 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bedb66f-11ea-42e1-a10c-3640bc51828b-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:36:03.812825 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:03.812774 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bedb66f-11ea-42e1-a10c-3640bc51828b-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:36:03.812825 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:03.812787 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4j2pk\" (UniqueName: \"kubernetes.io/projected/7bedb66f-11ea-42e1-a10c-3640bc51828b-kube-api-access-4j2pk\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:36:04.543142 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:04.543107 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7vnw9" Nov 03 21:36:04.543330 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:04.543142 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7vnw9" event={"ID":"7bedb66f-11ea-42e1-a10c-3640bc51828b","Type":"ContainerDied","Data":"7476c2bdda52f4fffa8812858bc73ad143362c081d1f5e73b48e9ffd4e85c3a8"} Nov 03 21:36:04.543330 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:04.543198 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7476c2bdda52f4fffa8812858bc73ad143362c081d1f5e73b48e9ffd4e85c3a8" Nov 03 21:36:12.869591 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:12.869543 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:36:16.321166 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:36:16.321127 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:36:19.068826 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:19.068763 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qkml7"] Nov 03 21:36:19.097571 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:19.092034 4305 cpu_manager.go:414] "containerMap: removing container" podUID="7bedb66f-11ea-42e1-a10c-3640bc51828b" containerName="revoke-keys" Nov 03 21:36:19.097571 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:19.092075 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bedb66f-11ea-42e1-a10c-3640bc51828b" containerName="revoke-keys" Nov 03 21:36:19.097571 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:19.092335 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bedb66f-11ea-42e1-a10c-3640bc51828b" containerName="revoke-keys" Nov 03 21:36:19.137366 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:19.137210 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qkml7"] Nov 03 21:36:19.138654 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:19.137612 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qkml7" Nov 03 21:36:19.195458 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:19.194773 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:36:19.244567 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:19.244531 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:36:19.244567 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:19.244557 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ftgz\"" Nov 03 21:36:19.244567 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:19.244542 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:36:19.309090 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:19.309036 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt5wm\" (UniqueName: \"kubernetes.io/projected/7989c54d-cc97-4429-807b-34cccc641309-kube-api-access-vt5wm\") pod \"maas-api-revoke-sub-0165d7d2-qkml7\" (UID: \"7989c54d-cc97-4429-807b-34cccc641309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qkml7" Nov 03 21:36:19.325529 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:19.324651 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7989c54d-cc97-4429-807b-34cccc641309-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-qkml7\" (UID: \"7989c54d-cc97-4429-807b-34cccc641309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qkml7" Nov 03 21:36:19.325529 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:19.324728 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7989c54d-cc97-4429-807b-34cccc641309-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-qkml7\" (UID: \"7989c54d-cc97-4429-807b-34cccc641309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qkml7" Nov 03 21:36:19.434684 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:19.430014 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt5wm\" (UniqueName: \"kubernetes.io/projected/7989c54d-cc97-4429-807b-34cccc641309-kube-api-access-vt5wm\") pod \"maas-api-revoke-sub-0165d7d2-qkml7\" (UID: \"7989c54d-cc97-4429-807b-34cccc641309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qkml7" Nov 03 21:36:19.434684 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:19.430112 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7989c54d-cc97-4429-807b-34cccc641309-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-qkml7\" (UID: \"7989c54d-cc97-4429-807b-34cccc641309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qkml7" Nov 03 21:36:19.434684 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:19.430150 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7989c54d-cc97-4429-807b-34cccc641309-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-qkml7\" (UID: \"7989c54d-cc97-4429-807b-34cccc641309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qkml7" Nov 03 21:36:19.434684 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:19.431043 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7989c54d-cc97-4429-807b-34cccc641309-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-qkml7\" (UID: \"7989c54d-cc97-4429-807b-34cccc641309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qkml7" Nov 03 21:36:19.445516 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:19.445419 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7989c54d-cc97-4429-807b-34cccc641309-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-qkml7\" (UID: \"7989c54d-cc97-4429-807b-34cccc641309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qkml7" Nov 03 21:36:19.498363 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:19.498312 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt5wm\" (UniqueName: \"kubernetes.io/projected/7989c54d-cc97-4429-807b-34cccc641309-kube-api-access-vt5wm\") pod \"maas-api-revoke-sub-0165d7d2-qkml7\" (UID: \"7989c54d-cc97-4429-807b-34cccc641309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qkml7" Nov 03 21:36:19.530065 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:19.530033 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qkml7" Nov 03 21:36:20.025551 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:20.024199 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qkml7"] Nov 03 21:36:20.643471 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:20.643378 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qkml7" event={"ID":"7989c54d-cc97-4429-807b-34cccc641309","Type":"ContainerStarted","Data":"08f29a4dcf68fea7752ee50b24ffc58cacddac7e1a2c2b473664763c05b9e2cf"} Nov 03 21:36:20.643471 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:20.643446 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qkml7" event={"ID":"7989c54d-cc97-4429-807b-34cccc641309","Type":"ContainerStarted","Data":"3d877c7767d6e264fbae3dbb87f64f8767d0b537ca79ce316fd296e01fac6ef3"} Nov 03 21:36:21.650422 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:21.650387 4305 generic.go:356] "Generic (PLEG): container finished" podID="7989c54d-cc97-4429-807b-34cccc641309" containerID="08f29a4dcf68fea7752ee50b24ffc58cacddac7e1a2c2b473664763c05b9e2cf" exitCode=0 Nov 03 21:36:21.650876 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:21.650480 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qkml7" event={"ID":"7989c54d-cc97-4429-807b-34cccc641309","Type":"ContainerDied","Data":"08f29a4dcf68fea7752ee50b24ffc58cacddac7e1a2c2b473664763c05b9e2cf"} Nov 03 21:36:22.798608 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:22.798583 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qkml7" Nov 03 21:36:22.969002 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:22.968899 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7989c54d-cc97-4429-807b-34cccc641309-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7989c54d-cc97-4429-807b-34cccc641309-maas-api-openshift-ca\") pod \"7989c54d-cc97-4429-807b-34cccc641309\" (UID: \"7989c54d-cc97-4429-807b-34cccc641309\") " Nov 03 21:36:22.969189 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:22.969026 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7989c54d-cc97-4429-807b-34cccc641309-kube-api-access-vt5wm\" (UniqueName: \"kubernetes.io/projected/7989c54d-cc97-4429-807b-34cccc641309-kube-api-access-vt5wm\") pod \"7989c54d-cc97-4429-807b-34cccc641309\" (UID: \"7989c54d-cc97-4429-807b-34cccc641309\") " Nov 03 21:36:22.969189 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:22.969078 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7989c54d-cc97-4429-807b-34cccc641309-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7989c54d-cc97-4429-807b-34cccc641309-maas-api-xks-ca\") pod \"7989c54d-cc97-4429-807b-34cccc641309\" (UID: \"7989c54d-cc97-4429-807b-34cccc641309\") " Nov 03 21:36:22.969345 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:22.969321 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7989c54d-cc97-4429-807b-34cccc641309-maas-api-openshift-ca" pod "7989c54d-cc97-4429-807b-34cccc641309" (UID: "7989c54d-cc97-4429-807b-34cccc641309"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:36:22.969449 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:22.969420 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7989c54d-cc97-4429-807b-34cccc641309-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:36:22.971285 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:22.971259 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7989c54d-cc97-4429-807b-34cccc641309-maas-api-xks-ca" pod "7989c54d-cc97-4429-807b-34cccc641309" (UID: "7989c54d-cc97-4429-807b-34cccc641309"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:36:22.971347 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:22.971315 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7989c54d-cc97-4429-807b-34cccc641309-kube-api-access-vt5wm" pod "7989c54d-cc97-4429-807b-34cccc641309" (UID: "7989c54d-cc97-4429-807b-34cccc641309"). InnerVolumeSpecName "kube-api-access-vt5wm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:36:23.070185 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:23.070144 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vt5wm\" (UniqueName: \"kubernetes.io/projected/7989c54d-cc97-4429-807b-34cccc641309-kube-api-access-vt5wm\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:36:23.070185 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:23.070180 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7989c54d-cc97-4429-807b-34cccc641309-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:36:23.663539 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:23.663504 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qkml7" event={"ID":"7989c54d-cc97-4429-807b-34cccc641309","Type":"ContainerDied","Data":"3d877c7767d6e264fbae3dbb87f64f8767d0b537ca79ce316fd296e01fac6ef3"} Nov 03 21:36:23.663843 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:23.663810 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d877c7767d6e264fbae3dbb87f64f8767d0b537ca79ce316fd296e01fac6ef3" Nov 03 21:36:23.664018 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:23.663899 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-qkml7" Nov 03 21:36:30.778978 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:30.778940 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9ljvt"] Nov 03 21:36:30.779645 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:30.779626 4305 cpu_manager.go:414] "containerMap: removing container" podUID="7989c54d-cc97-4429-807b-34cccc641309" containerName="revoke-keys" Nov 03 21:36:30.779741 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:30.779647 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7989c54d-cc97-4429-807b-34cccc641309" containerName="revoke-keys" Nov 03 21:36:30.779798 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:30.779748 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="7989c54d-cc97-4429-807b-34cccc641309" containerName="revoke-keys" Nov 03 21:36:30.787560 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:30.786721 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9ljvt" Nov 03 21:36:30.838833 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:30.838789 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t724s\" (UniqueName: \"kubernetes.io/projected/112b22ce-75fd-4237-9e74-770ef3ee5e96-kube-api-access-t724s\") pod \"maas-api-revoke-sub-6b19338f-9ljvt\" (UID: \"112b22ce-75fd-4237-9e74-770ef3ee5e96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9ljvt" Nov 03 21:36:30.839053 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:30.838866 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/112b22ce-75fd-4237-9e74-770ef3ee5e96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-9ljvt\" (UID: \"112b22ce-75fd-4237-9e74-770ef3ee5e96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9ljvt" Nov 03 21:36:30.839053 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:30.838933 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/112b22ce-75fd-4237-9e74-770ef3ee5e96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-9ljvt\" (UID: \"112b22ce-75fd-4237-9e74-770ef3ee5e96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9ljvt" Nov 03 21:36:30.840349 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:30.840317 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9ljvt"] Nov 03 21:36:30.886739 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:30.886709 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:36:30.886959 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:30.886846 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:36:30.886959 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:30.886866 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6ftgz\"" Nov 03 21:36:30.886959 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:30.886882 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:36:30.940289 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:30.940249 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/112b22ce-75fd-4237-9e74-770ef3ee5e96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-9ljvt\" (UID: \"112b22ce-75fd-4237-9e74-770ef3ee5e96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9ljvt" Nov 03 21:36:30.940528 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:30.940373 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t724s\" (UniqueName: \"kubernetes.io/projected/112b22ce-75fd-4237-9e74-770ef3ee5e96-kube-api-access-t724s\") pod \"maas-api-revoke-sub-6b19338f-9ljvt\" (UID: \"112b22ce-75fd-4237-9e74-770ef3ee5e96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9ljvt" Nov 03 21:36:30.940528 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:30.940422 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/112b22ce-75fd-4237-9e74-770ef3ee5e96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-9ljvt\" (UID: \"112b22ce-75fd-4237-9e74-770ef3ee5e96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9ljvt" Nov 03 21:36:30.941152 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:30.941127 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/112b22ce-75fd-4237-9e74-770ef3ee5e96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-9ljvt\" (UID: \"112b22ce-75fd-4237-9e74-770ef3ee5e96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9ljvt" Nov 03 21:36:30.942958 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:30.942933 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/112b22ce-75fd-4237-9e74-770ef3ee5e96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-9ljvt\" (UID: \"112b22ce-75fd-4237-9e74-770ef3ee5e96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9ljvt" Nov 03 21:36:31.001359 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:31.001329 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t724s\" (UniqueName: \"kubernetes.io/projected/112b22ce-75fd-4237-9e74-770ef3ee5e96-kube-api-access-t724s\") pod \"maas-api-revoke-sub-6b19338f-9ljvt\" (UID: \"112b22ce-75fd-4237-9e74-770ef3ee5e96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9ljvt" Nov 03 21:36:31.101802 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:31.101722 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9ljvt" Nov 03 21:36:31.487240 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:31.487210 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9ljvt"] Nov 03 21:36:31.489843 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:36:31.489801 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod112b22ce_75fd_4237_9e74_770ef3ee5e96.slice/crio-409fd4c3d8faaad46c9858bc9a07c7842751d5e33be881e6e313baad726ad214 WatchSource:0}: Error finding container 409fd4c3d8faaad46c9858bc9a07c7842751d5e33be881e6e313baad726ad214: Status 404 returned error can't find the container with id 409fd4c3d8faaad46c9858bc9a07c7842751d5e33be881e6e313baad726ad214 Nov 03 21:36:31.702784 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:31.702750 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9ljvt" event={"ID":"112b22ce-75fd-4237-9e74-770ef3ee5e96","Type":"ContainerStarted","Data":"409fd4c3d8faaad46c9858bc9a07c7842751d5e33be881e6e313baad726ad214"} Nov 03 21:36:32.708720 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:32.708625 4305 generic.go:356] "Generic (PLEG): container finished" podID="112b22ce-75fd-4237-9e74-770ef3ee5e96" containerID="42e11e287836e2c08e07c85e1c9202f3032400844cc501f2012d486caabd1b5e" exitCode=0 Nov 03 21:36:32.708720 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:32.708661 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9ljvt" event={"ID":"112b22ce-75fd-4237-9e74-770ef3ee5e96","Type":"ContainerDied","Data":"42e11e287836e2c08e07c85e1c9202f3032400844cc501f2012d486caabd1b5e"} Nov 03 21:36:33.875670 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:33.875637 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9ljvt" Nov 03 21:36:33.973073 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:33.972976 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/112b22ce-75fd-4237-9e74-770ef3ee5e96-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/112b22ce-75fd-4237-9e74-770ef3ee5e96-maas-api-openshift-ca\") pod \"112b22ce-75fd-4237-9e74-770ef3ee5e96\" (UID: \"112b22ce-75fd-4237-9e74-770ef3ee5e96\") " Nov 03 21:36:33.973248 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:33.973091 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/112b22ce-75fd-4237-9e74-770ef3ee5e96-kube-api-access-t724s\" (UniqueName: \"kubernetes.io/projected/112b22ce-75fd-4237-9e74-770ef3ee5e96-kube-api-access-t724s\") pod \"112b22ce-75fd-4237-9e74-770ef3ee5e96\" (UID: \"112b22ce-75fd-4237-9e74-770ef3ee5e96\") " Nov 03 21:36:33.973248 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:33.973152 4305 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/112b22ce-75fd-4237-9e74-770ef3ee5e96-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/112b22ce-75fd-4237-9e74-770ef3ee5e96-maas-api-xks-ca\") pod \"112b22ce-75fd-4237-9e74-770ef3ee5e96\" (UID: \"112b22ce-75fd-4237-9e74-770ef3ee5e96\") " Nov 03 21:36:33.973477 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:33.973413 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/112b22ce-75fd-4237-9e74-770ef3ee5e96-maas-api-openshift-ca" pod "112b22ce-75fd-4237-9e74-770ef3ee5e96" (UID: "112b22ce-75fd-4237-9e74-770ef3ee5e96"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:36:33.975336 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:33.975312 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/112b22ce-75fd-4237-9e74-770ef3ee5e96-maas-api-xks-ca" pod "112b22ce-75fd-4237-9e74-770ef3ee5e96" (UID: "112b22ce-75fd-4237-9e74-770ef3ee5e96"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:36:33.975424 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:33.975353 4305 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/112b22ce-75fd-4237-9e74-770ef3ee5e96-kube-api-access-t724s" pod "112b22ce-75fd-4237-9e74-770ef3ee5e96" (UID: "112b22ce-75fd-4237-9e74-770ef3ee5e96"). InnerVolumeSpecName "kube-api-access-t724s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:36:34.074957 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:34.074921 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/112b22ce-75fd-4237-9e74-770ef3ee5e96-maas-api-xks-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:36:34.074957 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:34.074951 4305 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/112b22ce-75fd-4237-9e74-770ef3ee5e96-maas-api-openshift-ca\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:36:34.074957 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:34.074961 4305 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t724s\" (UniqueName: \"kubernetes.io/projected/112b22ce-75fd-4237-9e74-770ef3ee5e96-kube-api-access-t724s\") on node \"ip-10-0-128-175.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 21:36:34.721005 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:34.720976 4305 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9ljvt" Nov 03 21:36:34.721005 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:34.721000 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-9ljvt" event={"ID":"112b22ce-75fd-4237-9e74-770ef3ee5e96","Type":"ContainerDied","Data":"409fd4c3d8faaad46c9858bc9a07c7842751d5e33be881e6e313baad726ad214"} Nov 03 21:36:34.721306 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:34.721044 4305 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="409fd4c3d8faaad46c9858bc9a07c7842751d5e33be881e6e313baad726ad214" Nov 03 21:36:43.708186 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:36:43.708148 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-9mqrd"] Nov 03 21:36:46.322658 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:36:46.322621 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:37:03.753717 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:37:03.753676 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6dc979975f-xfdzl_56e52b0e-8463-429a-8a83-6ce4d657d6e6/manager/0.log" Nov 03 21:37:04.120342 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:37:04.120273 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-wmqbg_b9977cd4-c16f-43e4-a79c-73af81749659/manager/0.log" Nov 03 21:37:04.355146 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:37:04.355110 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-w2wr7_67361d70-4a50-4746-a455-ded5894f3807/manager/0.log" Nov 03 21:37:07.560220 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:37:07.560144 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-qkml7_7989c54d-cc97-4429-807b-34cccc641309/revoke-keys/0.log" Nov 03 21:37:07.779964 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:37:07.779922 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-9ljvt_112b22ce-75fd-4237-9e74-770ef3ee5e96/revoke-keys/0.log" Nov 03 21:37:07.993620 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:37:07.993596 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-7vnw9_7bedb66f-11ea-42e1-a10c-3640bc51828b/revoke-keys/0.log" Nov 03 21:37:10.696412 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:37:10.696387 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-4xr8g_393b09a4-4340-4396-82fe-f91addfa5568/kuadrant-console-plugin/0.log" Nov 03 21:37:10.959752 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:37:10.959667 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-zffsq_c7af8a41-29dc-40bd-9065-074de2b99455/manager/0.log" Nov 03 21:37:11.309318 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:37:11.309247 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-9mqrd_58611e0c-31bc-46ca-b60e-a60632c5b273/limitador/0.log" Nov 03 21:37:12.346596 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:37:12.346564 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-5f555b57d98hbv2_621388a7-0f76-444c-b08b-83e93cf26d24/istio-proxy/0.log" Nov 03 21:37:13.040812 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:37:13.040777 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68d884f4c6-jxlpw_6cb31134-47e9-49b0-9412-e9dbcc5d588a/kube-auth-proxy/0.log" Nov 03 21:37:13.259310 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:37:13.259279 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n_5e88a59e-f72d-4345-8fc1-5c7113dc4638/istio-proxy/0.log" Nov 03 21:37:13.731602 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:37:13.731573 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz_9f872846-a625-48d9-af4e-5854d31069ab/istio-proxy/0.log" Nov 03 21:37:14.774218 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:37:14.774176 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7_308949ce-8a8c-4cd7-ac5a-a778aeaa4caf/istio-proxy/0.log" Nov 03 21:37:15.731477 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:37:15.731446 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr_148ae545-6008-4fd7-b538-eebc3a5e447f/istio-proxy/0.log" Nov 03 21:37:15.954907 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:37:15.954881 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-dctzh_d4d05666-5bea-415a-b266-805722b1158f/payload-pre-processing/0.log" Nov 03 21:37:16.179803 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:37:16.179769 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-z6z42_b6725e63-2f8d-4f1b-88ad-41057dd4a5c2/payload-processing/0.log" Nov 03 21:37:16.321817 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:37:16.321777 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:37:17.396200 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:37:17.396163 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-bblkx_d6ab5742-35d8-4f39-8faf-2cf2c209e5a2/main/0.log" Nov 03 21:37:17.652052 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:37:17.651971 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-c9n6g_5e264222-b7cf-4f08-85ff-f274be9562b5/main/0.log" Nov 03 21:37:17.877844 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:37:17.877811 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-jjtc9_8b0ee48b-a5e3-4989-8b7c-210a16e601be/main/0.log" Nov 03 21:37:18.091395 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:37:18.091306 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pblg_997eb6ae-52a1-4600-b955-59bdea0a6da3/main/0.log" Nov 03 21:37:18.777492 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:37:18.777426 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw_006c5eae-06af-447b-a1d8-cd719cce2a92/main/0.log" Nov 03 21:37:38.756868 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:37:38.756832 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6dc979975f-xfdzl_56e52b0e-8463-429a-8a83-6ce4d657d6e6/manager/0.log" Nov 03 21:37:39.132870 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:37:39.132774 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-wmqbg_b9977cd4-c16f-43e4-a79c-73af81749659/manager/0.log" Nov 03 21:37:39.361597 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:37:39.361566 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-w2wr7_67361d70-4a50-4746-a455-ded5894f3807/manager/0.log" Nov 03 21:37:46.321271 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:37:46.321233 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:38:02.442642 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:38:02.442609 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-qkml7_7989c54d-cc97-4429-807b-34cccc641309/revoke-keys/0.log" Nov 03 21:38:02.666181 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:38:02.666108 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-9ljvt_112b22ce-75fd-4237-9e74-770ef3ee5e96/revoke-keys/0.log" Nov 03 21:38:02.892654 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:38:02.892625 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-7vnw9_7bedb66f-11ea-42e1-a10c-3640bc51828b/revoke-keys/0.log" Nov 03 21:38:05.248265 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:38:05.248234 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-4xr8g_393b09a4-4340-4396-82fe-f91addfa5568/kuadrant-console-plugin/0.log" Nov 03 21:38:05.484983 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:38:05.484952 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-zffsq_c7af8a41-29dc-40bd-9065-074de2b99455/manager/0.log" Nov 03 21:38:05.759566 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:38:05.759535 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-9mqrd_58611e0c-31bc-46ca-b60e-a60632c5b273/limitador/0.log" Nov 03 21:38:06.574508 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:38:06.574472 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-5f555b57d98hbv2_621388a7-0f76-444c-b08b-83e93cf26d24/istio-proxy/0.log" Nov 03 21:38:07.257770 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:38:07.257737 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68d884f4c6-jxlpw_6cb31134-47e9-49b0-9412-e9dbcc5d588a/kube-auth-proxy/0.log" Nov 03 21:38:07.476443 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:38:07.476400 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-5ms2n_5e88a59e-f72d-4345-8fc1-5c7113dc4638/istio-proxy/0.log" Nov 03 21:38:07.918489 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:38:07.918452 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-7jgxz_9f872846-a625-48d9-af4e-5854d31069ab/istio-proxy/0.log" Nov 03 21:38:08.885892 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:38:08.885855 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-kwlc7_308949ce-8a8c-4cd7-ac5a-a778aeaa4caf/istio-proxy/0.log" Nov 03 21:38:09.805187 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:38:09.805158 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-c554b664c-4n9cr_148ae545-6008-4fd7-b538-eebc3a5e447f/istio-proxy/0.log" Nov 03 21:38:10.022795 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:38:10.022765 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-dctzh_d4d05666-5bea-415a-b266-805722b1158f/payload-pre-processing/0.log" Nov 03 21:38:10.247377 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:38:10.247342 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-z6z42_b6725e63-2f8d-4f1b-88ad-41057dd4a5c2/payload-processing/0.log" Nov 03 21:38:11.262368 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:38:11.262339 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-bblkx_d6ab5742-35d8-4f39-8faf-2cf2c209e5a2/main/0.log" Nov 03 21:38:11.534618 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:38:11.534524 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-c9n6g_5e264222-b7cf-4f08-85ff-f274be9562b5/main/0.log" Nov 03 21:38:11.758784 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:38:11.758755 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-jjtc9_8b0ee48b-a5e3-4989-8b7c-210a16e601be/main/0.log" Nov 03 21:38:11.971509 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:38:11.971468 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b88pblg_997eb6ae-52a1-4600-b955-59bdea0a6da3/main/0.log" Nov 03 21:38:12.633890 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:38:12.633849 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-5kmcw_006c5eae-06af-447b-a1d8-cd719cce2a92/main/0.log" Nov 03 21:38:16.322791 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:38:16.322754 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:38:46.320853 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:38:46.320806 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:39:14.154803 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:39:14.154775 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-z6z42_b6725e63-2f8d-4f1b-88ad-41057dd4a5c2/payload-processing/0.log" Nov 03 21:39:14.378549 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:39:14.378520 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-z6z42_b6725e63-2f8d-4f1b-88ad-41057dd4a5c2/payload-processing/0.log" Nov 03 21:39:16.322359 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:39:16.322320 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:39:33.193513 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:39:33.193471 4305 scope.go:122] "RemoveContainer" containerID="8ef6135fd9cf91c26f9982168649ba271acccad15d3e3236ab3c377e6a6b2dfc" Nov 03 21:39:46.322076 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:39:46.322037 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:40:16.322549 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:40:16.322511 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:40:24.750508 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:40:24.750469 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-z6z42_b6725e63-2f8d-4f1b-88ad-41057dd4a5c2/payload-processing/0.log" Nov 03 21:40:24.961655 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:40:24.961630 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-z6z42_b6725e63-2f8d-4f1b-88ad-41057dd4a5c2/payload-processing/0.log" Nov 03 21:40:34.722528 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:40:34.722496 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-g4892_03985fe9-0dfe-4f9a-97db-bb4c0298be5c/global-pull-secret-syncer/0.log" Nov 03 21:40:38.295133 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:40:38.295098 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-jvtt8_3e72f317-1042-40b0-b709-dc3d650754b5/konnectivity-agent/0.log" Nov 03 21:40:38.672078 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:40:38.672048 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-175.us-west-1.compute.internal_6873aa06186bc61b65aa0fa7f8c6d9a2/haproxy/0.log" Nov 03 21:40:44.141921 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:40:44.141887 4305 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fd9m6/perf-node-gather-daemonset-7xjzs"] Nov 03 21:40:44.142913 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:40:44.142885 4305 cpu_manager.go:414] "containerMap: removing container" podUID="112b22ce-75fd-4237-9e74-770ef3ee5e96" containerName="revoke-keys" Nov 03 21:40:44.142913 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:40:44.142909 4305 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="112b22ce-75fd-4237-9e74-770ef3ee5e96" containerName="revoke-keys" Nov 03 21:40:44.143081 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:40:44.143003 4305 memory_manager.go:365] "RemoveStaleState removing state" podUID="112b22ce-75fd-4237-9e74-770ef3ee5e96" containerName="revoke-keys" Nov 03 21:40:44.146233 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:40:44.146215 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fd9m6/perf-node-gather-daemonset-7xjzs" Nov 03 21:40:44.199814 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:40:44.199732 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fd9m6\"/\"kube-root-ca.crt\"" Nov 03 21:40:44.243615 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:40:44.243579 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/aeb92c98-4a93-4940-8bfc-34a3a6a1eec6-podres\") pod \"perf-node-gather-daemonset-7xjzs\" (UID: \"aeb92c98-4a93-4940-8bfc-34a3a6a1eec6\") " pod="openshift-must-gather-fd9m6/perf-node-gather-daemonset-7xjzs" Nov 03 21:40:44.243822 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:40:44.243672 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aeb92c98-4a93-4940-8bfc-34a3a6a1eec6-sys\") pod \"perf-node-gather-daemonset-7xjzs\" (UID: \"aeb92c98-4a93-4940-8bfc-34a3a6a1eec6\") " pod="openshift-must-gather-fd9m6/perf-node-gather-daemonset-7xjzs" Nov 03 21:40:44.243822 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:40:44.243817 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aeb92c98-4a93-4940-8bfc-34a3a6a1eec6-lib-modules\") pod \"perf-node-gather-daemonset-7xjzs\" (UID: \"aeb92c98-4a93-4940-8bfc-34a3a6a1eec6\") " pod="openshift-must-gather-fd9m6/perf-node-gather-daemonset-7xjzs" Nov 03 21:40:44.243957 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:40:44.243853 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/aeb92c98-4a93-4940-8bfc-34a3a6a1eec6-proc\") pod \"perf-node-gather-daemonset-7xjzs\" (UID: \"aeb92c98-4a93-4940-8bfc-34a3a6a1eec6\") " pod="openshift-must-gather-fd9m6/perf-node-gather-daemonset-7xjzs" Nov 03 21:40:44.243957 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:40:44.243878 4305 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq56w\" (UniqueName: \"kubernetes.io/projected/aeb92c98-4a93-4940-8bfc-34a3a6a1eec6-kube-api-access-tq56w\") pod \"perf-node-gather-daemonset-7xjzs\" (UID: \"aeb92c98-4a93-4940-8bfc-34a3a6a1eec6\") " pod="openshift-must-gather-fd9m6/perf-node-gather-daemonset-7xjzs" Nov 03 21:40:44.249662 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:40:44.249635 4305 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fd9m6\"/\"openshift-service-ca.crt\"" Nov 03 21:40:44.249897 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:40:44.249878 4305 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fd9m6\"/\"default-dockercfg-9chns\"" Nov 03 21:40:44.253300 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:40:44.253272 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fd9m6/perf-node-gather-daemonset-7xjzs"] Nov 03 21:40:44.344988 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:40:44.344943 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aeb92c98-4a93-4940-8bfc-34a3a6a1eec6-lib-modules\") pod \"perf-node-gather-daemonset-7xjzs\" (UID: \"aeb92c98-4a93-4940-8bfc-34a3a6a1eec6\") " pod="openshift-must-gather-fd9m6/perf-node-gather-daemonset-7xjzs" Nov 03 21:40:44.345200 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:40:44.345002 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/aeb92c98-4a93-4940-8bfc-34a3a6a1eec6-proc\") pod \"perf-node-gather-daemonset-7xjzs\" (UID: \"aeb92c98-4a93-4940-8bfc-34a3a6a1eec6\") " pod="openshift-must-gather-fd9m6/perf-node-gather-daemonset-7xjzs" Nov 03 21:40:44.345200 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:40:44.345030 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq56w\" (UniqueName: \"kubernetes.io/projected/aeb92c98-4a93-4940-8bfc-34a3a6a1eec6-kube-api-access-tq56w\") pod \"perf-node-gather-daemonset-7xjzs\" (UID: \"aeb92c98-4a93-4940-8bfc-34a3a6a1eec6\") " pod="openshift-must-gather-fd9m6/perf-node-gather-daemonset-7xjzs" Nov 03 21:40:44.345200 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:40:44.345062 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/aeb92c98-4a93-4940-8bfc-34a3a6a1eec6-podres\") pod \"perf-node-gather-daemonset-7xjzs\" (UID: \"aeb92c98-4a93-4940-8bfc-34a3a6a1eec6\") " pod="openshift-must-gather-fd9m6/perf-node-gather-daemonset-7xjzs" Nov 03 21:40:44.345200 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:40:44.345095 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aeb92c98-4a93-4940-8bfc-34a3a6a1eec6-lib-modules\") pod \"perf-node-gather-daemonset-7xjzs\" (UID: \"aeb92c98-4a93-4940-8bfc-34a3a6a1eec6\") " pod="openshift-must-gather-fd9m6/perf-node-gather-daemonset-7xjzs" Nov 03 21:40:44.345200 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:40:44.345100 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/aeb92c98-4a93-4940-8bfc-34a3a6a1eec6-proc\") pod \"perf-node-gather-daemonset-7xjzs\" (UID: \"aeb92c98-4a93-4940-8bfc-34a3a6a1eec6\") " pod="openshift-must-gather-fd9m6/perf-node-gather-daemonset-7xjzs" Nov 03 21:40:44.345200 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:40:44.345133 4305 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aeb92c98-4a93-4940-8bfc-34a3a6a1eec6-sys\") pod \"perf-node-gather-daemonset-7xjzs\" (UID: \"aeb92c98-4a93-4940-8bfc-34a3a6a1eec6\") " pod="openshift-must-gather-fd9m6/perf-node-gather-daemonset-7xjzs" Nov 03 21:40:44.345200 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:40:44.345167 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/aeb92c98-4a93-4940-8bfc-34a3a6a1eec6-podres\") pod \"perf-node-gather-daemonset-7xjzs\" (UID: \"aeb92c98-4a93-4940-8bfc-34a3a6a1eec6\") " pod="openshift-must-gather-fd9m6/perf-node-gather-daemonset-7xjzs" Nov 03 21:40:44.345545 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:40:44.345223 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aeb92c98-4a93-4940-8bfc-34a3a6a1eec6-sys\") pod \"perf-node-gather-daemonset-7xjzs\" (UID: \"aeb92c98-4a93-4940-8bfc-34a3a6a1eec6\") " pod="openshift-must-gather-fd9m6/perf-node-gather-daemonset-7xjzs" Nov 03 21:40:44.406398 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:40:44.406363 4305 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq56w\" (UniqueName: \"kubernetes.io/projected/aeb92c98-4a93-4940-8bfc-34a3a6a1eec6-kube-api-access-tq56w\") pod \"perf-node-gather-daemonset-7xjzs\" (UID: \"aeb92c98-4a93-4940-8bfc-34a3a6a1eec6\") " pod="openshift-must-gather-fd9m6/perf-node-gather-daemonset-7xjzs" Nov 03 21:40:44.459681 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:40:44.459650 4305 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fd9m6/perf-node-gather-daemonset-7xjzs" Nov 03 21:40:44.627511 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:40:44.627477 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-4xr8g_393b09a4-4340-4396-82fe-f91addfa5568/kuadrant-console-plugin/0.log" Nov 03 21:40:44.833466 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:40:44.833417 4305 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fd9m6/perf-node-gather-daemonset-7xjzs"] Nov 03 21:40:44.835229 ip-10-0-128-175 kubenswrapper[4305]: W1103 21:40:44.835201 4305 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podaeb92c98_4a93_4940_8bfc_34a3a6a1eec6.slice/crio-3a7973aeb459160b0fb4a2f2331f9c7fcd82bea2dd59582db2d77191ac74eafb WatchSource:0}: Error finding container 3a7973aeb459160b0fb4a2f2331f9c7fcd82bea2dd59582db2d77191ac74eafb: Status 404 returned error can't find the container with id 3a7973aeb459160b0fb4a2f2331f9c7fcd82bea2dd59582db2d77191ac74eafb Nov 03 21:40:45.013124 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:40:45.013081 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fd9m6/perf-node-gather-daemonset-7xjzs" event={"ID":"aeb92c98-4a93-4940-8bfc-34a3a6a1eec6","Type":"ContainerStarted","Data":"42e616ba7e5e38996f274e87f6b0229e8d491c279f9d90cd5c4d89ed205ab5f7"} Nov 03 21:40:45.013331 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:40:45.013147 4305 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fd9m6/perf-node-gather-daemonset-7xjzs" event={"ID":"aeb92c98-4a93-4940-8bfc-34a3a6a1eec6","Type":"ContainerStarted","Data":"3a7973aeb459160b0fb4a2f2331f9c7fcd82bea2dd59582db2d77191ac74eafb"} Nov 03 21:40:45.013331 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:40:45.013269 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-fd9m6/perf-node-gather-daemonset-7xjzs" Nov 03 21:40:45.077867 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:40:45.077826 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-zffsq_c7af8a41-29dc-40bd-9065-074de2b99455/manager/0.log" Nov 03 21:40:45.131364 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:40:45.131262 4305 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-fd9m6/perf-node-gather-daemonset-7xjzs" podStartSLOduration=1.131246769 podStartE2EDuration="1.131246769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:40:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:40:45.130532684 +0000 UTC m=+3680.355852477" watchObservedRunningTime="2026-11-03 21:40:45.131246769 +0000 UTC m=+3680.356566560" Nov 03 21:40:45.466033 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:40:45.465997 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-9mqrd_58611e0c-31bc-46ca-b60e-a60632c5b273/limitador/0.log" Nov 03 21:40:46.322368 ip-10-0-128-175 kubenswrapper[4305]: E1103 21:40:46.322333 4305 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b4gn6" containerName="extractor" Nov 03 21:40:51.028965 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:40:51.028939 4305 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-fd9m6/perf-node-gather-daemonset-7xjzs" Nov 03 21:40:51.959899 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:40:51.959866 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-tv9dk_8f54b118-294c-48cf-9bd7-20a2602043d2/kube-state-metrics/0.log" Nov 03 21:40:52.368399 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:40:52.368310 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-tv9dk_8f54b118-294c-48cf-9bd7-20a2602043d2/kube-rbac-proxy-main/0.log" Nov 03 21:40:52.786270 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:40:52.786240 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-tv9dk_8f54b118-294c-48cf-9bd7-20a2602043d2/kube-rbac-proxy-self/0.log" Nov 03 21:40:53.207823 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:40:53.207790 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7d9ffb76d9-hvc98_7fae9b94-1aeb-4bfd-b435-4c9397907762/metrics-server/0.log" Nov 03 21:40:53.626260 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:40:53.626176 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-tjq2x_98aa29ea-e74b-41ce-ab99-18b3b7e84338/monitoring-plugin/0.log" Nov 03 21:40:54.049144 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:40:54.049114 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2s79k_ac7e0d69-b7d1-4b23-9526-71e488829501/node-exporter/0.log" Nov 03 21:40:54.463710 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:40:54.463676 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2s79k_ac7e0d69-b7d1-4b23-9526-71e488829501/kube-rbac-proxy/0.log" Nov 03 21:40:54.882992 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:40:54.882921 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2s79k_ac7e0d69-b7d1-4b23-9526-71e488829501/init-textfile/0.log" Nov 03 21:40:57.811663 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:40:57.811630 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-xsn6d_592c07dc-3fef-4f51-87a4-669769c5a9ef/kube-rbac-proxy-main/0.log" Nov 03 21:40:58.231657 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:40:58.231611 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-xsn6d_592c07dc-3fef-4f51-87a4-669769c5a9ef/kube-rbac-proxy-self/0.log" Nov 03 21:40:58.649518 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:40:58.649485 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-xsn6d_592c07dc-3fef-4f51-87a4-669769c5a9ef/openshift-state-metrics/0.log" Nov 03 21:40:59.075640 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:40:59.075564 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_aa9b3422-1602-42a0-a7ec-a3b520501772/prometheus/0.log" Nov 03 21:40:59.490423 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:40:59.490388 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_aa9b3422-1602-42a0-a7ec-a3b520501772/config-reloader/0.log" Nov 03 21:40:59.906824 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:40:59.906798 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_aa9b3422-1602-42a0-a7ec-a3b520501772/thanos-sidecar/0.log" Nov 03 21:41:00.322501 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:41:00.322400 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_aa9b3422-1602-42a0-a7ec-a3b520501772/kube-rbac-proxy-web/0.log" Nov 03 21:41:00.741449 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:41:00.741408 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_aa9b3422-1602-42a0-a7ec-a3b520501772/kube-rbac-proxy/0.log" Nov 03 21:41:01.170128 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:41:01.170102 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_aa9b3422-1602-42a0-a7ec-a3b520501772/kube-rbac-proxy-thanos/0.log" Nov 03 21:41:01.629971 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:41:01.629900 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_aa9b3422-1602-42a0-a7ec-a3b520501772/init-config-reloader/0.log" Nov 03 21:41:03.046472 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:41:03.046415 4305 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7vnw9"] Nov 03 21:41:03.095635 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:41:03.095601 4305 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-7vnw9"] Nov 03 21:41:03.330770 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:41:03.330691 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-574598d969-4hlgw_2e75299b-366a-4617-8c73-a12733d8db8c/telemeter-client/0.log" Nov 03 21:41:03.746346 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:41:03.746317 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-574598d969-4hlgw_2e75299b-366a-4617-8c73-a12733d8db8c/reload/0.log" Nov 03 21:41:04.161489 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:41:04.161461 4305 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-574598d969-4hlgw_2e75299b-366a-4617-8c73-a12733d8db8c/kube-rbac-proxy/0.log" Nov 03 21:41:04.894535 ip-10-0-128-175 kubenswrapper[4305]: I1103 21:41:04.894501 4305 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bedb66f-11ea-42e1-a10c-3640bc51828b" path="/var/lib/kubelet/pods/7bedb66f-11ea-42e1-a10c-3640bc51828b/volumes"